Communication system, display device, display control method, and display control program

Abstract

To provide a communication system, a display device, a display control method, and a display control program, which allow for presenting appropriate visual information associated with information transmission along with the progress of information transmission.SOLUTION: A communication system provided herein comprises: linguistic information input means for receiving input of linguistic information; recognition means for recognizing the input linguistic information; and image display means configured to display an image corresponding to the input linguistic information based on a recognition result of the recognition means on display means for displaying images.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] The present invention relates to a communication system, a display device, a display control method, and a display control program. [Background technology]

[0002] In communication situations, verbal communication alone is sometimes insufficient to transmit information efficiently in a short amount of time. For example, in meetings where new plans or ideas are being considered, it is effective to utilize visual information for communication.

[0003] Patent Document 1 describes a system in which an illustration displayed in an illustration search result display area on a minutes editing screen of a graphic recording system is selected, and the selected illustration is pasted into the minutes to create minutes that incorporate illustrations. It describes how viewing the minutes that incorporate illustrations allows for an efficient review of the meeting. Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, illustrations are searched for selected text, and multiple illustrations are displayed in an illustration search result display area for the text.

[0005] The present invention has been made in consideration of the above, and aims to provide a communication system, a display device, a display control method, and a display control program that can present appropriate visual information related to information transmission as information transmission progresses via voice, text input, etc. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, a communication system according to one aspect of the present invention includes a language information input unit that accepts input of language information, and a language information input unit that converts the input language information into a As text information a recognition means for recognizing the image based on the recognition result of the recognition means; Keywords contained in the text information an image display means for displaying an image corresponding to the image on a display means for displaying an image; a selection means for selecting the displayed image; and another image display means for displaying an image corresponding to the selected image on the display means in a state in which an editing process operation by a user can be accepted. Equipped with. [Effects of the Invention]

[0007] According to the present invention, it is possible to present appropriate visual information related to information transmission as information transmission by voice, text input, or the like progresses. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a system configuration diagram showing the system configuration of a visual communication system according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a functional configuration of the visual communication system according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating a hardware configuration of a computer applied to the visual communication system according to the embodiment. [Figure 4] FIG. 4 is a diagram showing the configuration of the language information input unit in the embodiment. [Figure 5] FIG. 5 is a diagram illustrating a configuration of a screen display control unit in the embodiment. [Figure 6] FIG. 6 is a data flow diagram illustrating the conversion of image data in an embodiment. [Figure 7] FIG. 7 is a diagram showing the data structure of illustration information stored in the illustration information storage unit in the embodiment. [Figure 8] FIG. 8 is a diagram showing the data structure of thumbnail information stored in the thumbnail information storage unit in the embodiment. [Figure 9]FIG. 9 is a diagram showing a modified example of the configuration of the screen display control unit in the embodiment. [Figure 10] FIG. 10 is a diagram showing the data structure of thumbnail information stored in the thumbnail information storage unit. [Figure 11] FIG. 11 is a functional block diagram of a priority image presenting unit in the embodiment. [Figure 12] FIG. 12 is a diagram showing the configuration of the drawing operation unit in the embodiment. [Figure 13] FIG. 13 is a flowchart illustrating the operation of the visual communication system according to the embodiment. [Figure 14] FIG. 14 is a flowchart showing the flow of the registration process in the embodiment. [Figure 15] FIG. 15 is a flowchart showing the flow of the communication support process in the embodiment. [Figure 16] FIG. 16 is a flowchart showing the flow of the communication support process in the embodiment. [Figure 17] FIG. 17 is a diagram showing the configuration of a display screen of the visual communication system according to the embodiment. [Figure 18] FIG. 18 is a diagram showing a schematic operation of a display screen of the visual communication system according to the embodiment. [Figure 19] FIG. 19 is a diagram showing the flow of two-stage editing processing in the embodiment. [Figure 20] FIG. 20 is a diagram showing detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 21] FIG. 21 is a diagram showing detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 22] FIG. 22 is a diagram showing detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 23]FIG. 23 is a diagram showing detailed operations of a display screen by the visual communication system in the embodiment. [Figure 24-1] FIG. 24A is a diagram illustrating detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 24-2] FIG. 24B is a diagram illustrating detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 24-3] FIG. 24C is a diagram illustrating detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 25] FIG. 25 is a diagram showing detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 26] FIG. 26 is a diagram showing detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 27] FIG. 27 is a diagram showing detailed operations of a display screen by the visual communication system according to the embodiment. [Figure 28] FIG. 28 is a system configuration diagram illustrating a system configuration of an online conference system according to a modified example of the embodiment. [Figure 29] FIG. 29 is a block diagram illustrating a functional configuration of an online conference system according to a modified example of the embodiment. [Figure 30] FIG. 30 is a diagram showing an example of a usage scene in which the online conference system is applied to a telepresence robot. [Figure 31] FIG. 31 is a diagram showing a camera image captured by a camera of a client terminal in a modified example of the embodiment. [Figure 32] FIG. 32 is a diagram showing an image superimposed on the camera image of FIG. 31 by a drawing information generating unit in a modification of the embodiment. [Figure 33] FIG. 33 is a diagram showing an image displayed on another client terminal in the modified example of the embodiment. [Figure 34]FIG. 34 is a diagram showing an image of a gesture on the drawing result display unit in the modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment) A visual communication system according to an embodiment has a function of supporting communication. In communication situations, communication based on linguistic information alone may not be sufficient to efficiently transmit information in a short amount of time. For example, in a meeting held to consider new plans or ideas, the use of visual information is effective for communication. While sketching an idea by hand is one way to visualize it, it is sometimes not easy to visualize the image in one's head from scratch. Therefore, a first technology has been considered that converts linguistic information, which is easy for anyone to use, into visual information.

[0010] However, the first technology, which converts linguistic information into visual information, merely searches for pre-prepared illustrations when a user selects linguistic information. With the first technology, there are often cases where the orientation or combination of illustrations does not match the image the user wants to convey, making it difficult to properly communicate information. Increasing the number of pre-prepared illustrations increases the probability that an illustration matching the image in one's mind will be found, but the time required for selection increases as the number of illustrations increases. Furthermore, when multiple pieces of linguistic information are selected, illustrations are searched for and displayed for each piece of linguistic information. Therefore, when the first technology is applied to real-time communication situations, illustrations are displayed for linguistic information other than the currently spoken or input linguistic information, which can make illustration selection time-consuming. In other words, the first technology may not be able to quickly communicate information in communication situations such as meetings.

[0011] Therefore, in this embodiment, in a visual communication system, one or more thumbnail images that are candidates for illustrations corresponding to language information are quickly switched and displayed in response to changes in the language information, thereby enabling the rapid selection of an illustration that corresponds to the current language information and supporting the rapid transmission of information in communication situations.

[0012] Specifically, a visual communication system converts linguistic information into editable visual information, enabling quick and accurate communication of mental images during communication situations such as meetings. The visual communication system analyzes linguistic information, such as speech and text input, in real time during communication situations, and simultaneously displays one or more thumbnail images in a specific area as candidate illustrations related to the linguistic information. The visual communication system has first correspondence information in which linguistic information and thumbnail images are associated. Upon receiving linguistic information, the visual communication system identifies and displays one or more thumbnail images corresponding to the linguistic information in the first correspondence information. The visual communication system also has second correspondence information in which thumbnail images are associated with illustrations. By converting thumbnail images into two-dimensional images when illustrations are three-dimensional, the visual communication system can display thumbnail images more quickly than when displaying illustrations. Each time linguistic information from a user is received, the thumbnail images on the display screen can be quickly updated and displayed. This allows a user to select one or more thumbnail images that best represent the mental image they had when the speech or text input was used, without interrupting communication through conversation or other means. According to the second correspondence information, the 3D illustration data associated with the selected thumbnail image is searched for, and the identified 3D illustration data is displayed. This allows for three-dimensional editing (three-dimensional movement, scaling, and rotation) of the 3D illustration data. The visual communication system then converts (fixes) the 3D illustration data, which can be edited in three dimensions, into two-dimensional illustration data, which can be edited in two dimensions. Accordingly, two-dimensional editing (freely erasing parts, adding on top, or painting) of the 2D illustration data becomes possible. This rapid switching of thumbnail images triggered by automatic recognition of language information and the corresponding two-stage editing process allow for quick and accurate expression of the image in one's mind in real-time communication situations, enabling rapid and accurate communication.

[0013] More specifically, the visual communication system 4 can be configured as shown in Fig. 1. Fig. 1 is a system configuration diagram showing the system configuration of the visual communication system 4.

[0014] The visual communication system 4 includes a client terminal 1, a server 2, and a connection unit 3. The connection unit 3 connects the client terminal 1 and the server 2 so that they can communicate with each other.

[0015] 2 is a block diagram showing the functional configuration of the visual communication system 4. The client terminal 1 has a language information input unit 100, a screen display control unit 200, and a drawing operation unit 300. The server 2 has a drawing information generation unit 400 and a memory unit 500. The memory unit 500 stores a program 500a.

[0016] When the visual communication system 4 receives a startup request from a user at the client terminal 1, the startup request is transmitted from the client terminal 1 to the server 2, and in response to the startup request, the server 2 reads out the program 500a from the storage unit 500. In accordance with the program 500a, the visual communication system 4 functionally configures a language information input unit 100, a screen display control unit 200, and a drawing operation unit 300 in the client terminal 1, and functionally configures a drawing information generation unit 400 in the server 2, as shown in FIG.

[0017] In the visual communication system 4, the screen display control unit 200 and the drawing operation unit 300 in the client terminal 1 may be configured in the server 2 or another server. Alternatively, the visual communication system 4 may be configured so that all functional components including the language information input unit 100, the screen display control unit 200, the drawing operation unit 300, and the drawing information generation unit 400 are completed within the client terminal 1. Alternatively, the visual communication system 4 may be configured so that some of the elements included in the screen display control unit 200 (e.g., the user interface and similar parts) are configured within the client terminal 1, and the remaining parts are configured within the server 2 or another server. Similarly, the visual communication system 4 may be configured so that some of the elements included in the drawing operation unit 300 (e.g., the user interface and similar parts) are configured within the client terminal 1, and the remaining parts are configured within the server 2 or another server.

[0018] The connection unit 3 may be a wired communication line and / or a wireless communication line, may be a so-called communication network, or may be a communication cable, etc. The connection unit 3 may be constructed using one or more of the Internet, a mobile communication network, a LAN (Local Area Network), etc. The connection unit 3 may include not only wired communication but also a wireless communication network such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution). When the program 500a is stored in the client terminal 1 and each functional configuration is configured to be completed within the client terminal 1, the connection unit 3 may be omitted.

[0019] In the client terminal 1, the language information input unit 100, which serves as a language information input means, accepts input of language information by the user. The screen display control unit 200 displays images such as characters and illustrations on the display 506 (see FIG. 3). The drawing operation unit 300 accepts drawing operations by the user. Here, drawing includes not only handwriting but also selection operations on the display 506 for the purpose of drawing on the display 506. The drawing operation unit 300 of the client terminal 1 transmits a drawing operation request to the drawing information generation unit 400 of the server 2. In response to the drawing operation request, the drawing information generation unit 400 of the server 2 updates the drawing information so as to change the display form of the displayed image and transmits the updated drawing information to the client terminal 1. The client terminal 1 receives the updated drawing information and displays an image corresponding to the updated drawing information on the display 506. As a result, the result of the drawing operation by the user is displayed on the display 506 of the client terminal 1.

[0020] The client terminal 1 and the server 2 may each be configured in hardware as a computer 5 as shown in Fig. 3. Fig. 3 is a diagram showing the hardware configuration of the computer 5 applied to the visual communication system 4.

[0021] As shown in FIG. 3, the computer 5 includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, a media I / F 516, a motion acquisition device 517, a microphone 518, a speaker 519, and a camera 520.

[0022] Of these, the CPU 501 controls the overall operation of the computer 5. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as the program 500a. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506, which serves as a display means, displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories, printers, etc. The network I / F 509 is an interface for data communication using the connection unit 3. The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components, such as the CPU 501, shown in FIG. 3.

[0023] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0024] The motion acquisition device 517 is a circuit that detects the user's motion and converts it into an electrical signal, and is a type of input means. The user's motion may be detected by any of the following means: an optical method that detects light reflection; a magnetic sensor, a mechanical sensor, or a magnetic sensor attached to the user; a method that analyzes an image of the user; or a combination of these.

[0025] The microphone 518 is a built-in circuit that converts sound into an electrical signal. The speaker 519 is a built-in circuit that converts the electrical signal into physical vibrations to produce sounds such as music and voice.

[0026] The speaker 519 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice.

[0027] The camera 520 is a type of built-in imaging means that captures an image of a subject and obtains image data.

[0028] In the computer 5 applied to the server 2, the HD 504 corresponds to the storage unit 500 and stores the program 500a. In the computer 5 applied to the client terminal 1, the program 500a is downloaded from the server 2 via the connection unit 3 and stored in the HD 504, and the CPU 501 reads and executes the program 500a from the HD 504, thereby allowing the functional configuration as shown in Fig. 2 to be deployed in the RAM 503 either collectively at the time of compilation or sequentially as the processing progresses. In the computer 5 applied to the server 2, the program 500a is executed by the CPU 501, allowing the functional configuration as shown in Fig. 2 to be deployed in the RAM 503 either collectively at the time of compilation or sequentially as the processing progresses.

[0029] Furthermore, the computer 5 applied to the server 2 may omit the configuration related to the user interface, and at least one of the display 506, keyboard 511, pointing device 512, and action acquisition device 517 may be omitted.

[0030] In addition to the computer 5, the client terminal 1 and the server 2 may also be output devices such as an IWB (Interactive White Board: an electronic white board with a blackboard function that allows mutual communication), digital signage, a HUD (Head Up Display) device, industrial machinery, medical equipment, network home appliances, automobiles (Connected Cars), mobile phones, smartphones, tablet terminals, game consoles, PDAs (Personal Digital Assistants), etc.

[0031] The language information input unit 100 shown in Fig. 2 can be functionally configured as shown in Fig. 4. Fig. 4 is a diagram showing the configuration of the language information input unit 100.

[0032] The language information input unit 100 includes a voice input unit 101, a character input unit 102, a recognition unit 103 as a recognition means, and a recognition result storage unit 105. Each of these units is a function or means realized by operating in accordance with an instruction from a CPU 501 in accordance with a program loaded from an HD 504 onto a RAM 503.

[0033] The voice input unit 101 receives, as language information, a voice signal converted from the user's voice by the microphone 518 or a voice signal received via the network I / F 509. The voice input unit 101 supplies the voice signal to the recognition unit 103.

[0034] The character input unit 102 accepts, as language information, character signals input by the user via the keyboard 511, pointing device 512, or action acquisition device 517, or character signals received via a network I / F. The character input unit 102 supplies the character signals to the recognition unit 103. It is assumed that characters are input to the character input unit 102 by typing on the keyboard 511 or by handwriting input using the pointing device 512 or action acquisition device 517.

[0035] Upon receiving linguistic information from the speech input unit 101 or the character input unit 102, the recognition unit 103 recognizes character information from the linguistic information. The recognition unit 103 stores the recognition result in the recognition result storage unit 105.

[0036] When the recognition unit 103 receives a voice signal from the voice input unit 101, it performs voice recognition processing on the voice signal and converts it into text information. The recognition unit 103 has a template voice signal for each character. The recognition unit 103 performs matching processing on the voice signal using the template voice signal, and can recognize characters for each voice based on the matching score. As a result, the recognition unit 103 generates text information as the recognition result for the voice signal.

[0037] When the recognition unit 103 receives a character signal from the character input unit 102, it generates character information as the recognition result for the character signal. When the recognition unit 103 receives character information typed on the keyboard 511, it treats the character information as the recognition result. When the recognition unit 103 receives a handwritten character image using the pointing device 512 or the motion acquisition device 517, it performs text recognition processing on the handwritten character image and converts it into character information. The recognition unit 103 has a template character image for each character. The recognition unit 103 performs a matching process on the handwritten character image using the template character image, and can recognize characters for each handwritten character image based on the matching score. As a result, the recognition unit 103 generates character information as the recognition result for the handwritten character image. The recognition by the recognition unit 103 is not limited to this; for example, the recognition unit 103 may generate a characteristic waveform pattern contained in the input audio signal as the recognition result.

[0038] The recognition unit 103 transmits the recognition result stored in the recognition result storage unit 105 to the screen display control unit 200 .

[0039] The screen display control unit 200 shown in Fig. 2 can be functionally configured as shown in Fig. 5. Fig. 5 is a diagram showing the configuration of the screen display control unit 200.

[0040] The screen display control unit 200 includes a thumbnail search unit 202 as a first image search means, a thumbnail display unit 203 as a first image display means, a thumbnail selection unit 204, an illustration search unit 205 as a second image search means, an illustration display unit 206 as a second image display means, a priority image presentation unit 220 as a priority image presentation means, a thumbnail generation unit 211, an illustration generation unit 213, a 3D model input unit 214, a keyword generation unit 215, and a registered character input unit 216. The priority image presentation unit 220 includes a context analysis unit 207, a priority determination unit 208, and a selection tendency accumulation unit 209.

[0041] These units are functions or means that are realized by operating in accordance with instructions from the CPU 501 in accordance with the program 500 a loaded from the HD 504 onto the RAM 503 .

[0042] The screen display control unit 200 also includes a thumbnail information storage unit 210, which is a first storage unit, and an illustration information storage unit 212, which is a second storage unit. Each of these units is constructed using the ROM 502, the RAM 503, or the HD 504.

[0043] The storage of data in thumbnail information storage unit 210 and illustration information storage unit 212 will be described with reference to Figures 5 and 6. Figure 6 is a data flow diagram showing the conversion of image data.

[0044] The three-dimensional model input unit 214 receives three-dimensional model data. The three-dimensional model data is, for example, polygon data as shown in FIG. 6(a) and includes a plurality of spatial coordinates. The three-dimensional model data corresponds to a three-dimensional image data format (for example, OBJ format). The three-dimensional model input unit 214 supplies the three-dimensional model data to the illustration unit 213.

[0045] The illustration generating unit 213 converts the three-dimensional model data into three-dimensional illustration data. The three-dimensional illustration data is, for example, three-dimensional line drawing data as shown in FIG. 6(b) and includes multiple spatial coordinates. The three-dimensional illustration data corresponds to a three-dimensional image data format (for example, the OBJ format). The illustration generating unit 213 extracts line drawing information from the three-dimensional model data by identifying the three-dimensional positions of the edges of the three-dimensional shape represented by the three-dimensional model data and connecting them with line drawings, thereby generating three-dimensional illustration data. The illustration generating unit 213 additionally stores the three-dimensional illustration data in the illustration information storage unit 212.

[0046] As a result, illustration information 212a as shown in FIG. 7 is stored in illustration information storage unit 212. FIG. 7 is a diagram showing the data structure of illustration information 212a stored in illustration information storage unit 212. In illustration information 212a, 3D illustration data and its identification information are associated with one or more pieces of 3D illustration data. For example, illustration information 212a has an identification information field 212a1 and an access information field 212a2. Information for identifying the 3D illustration data, such as an ID number of the 3D illustration data, is recorded in identification information field 212a1. Information for accessing the 3D illustration data, such as a file name of the 3D illustration data, is recorded in access information field 212a2. The information for accessing the 3D illustration data is not limited to a file name, and may be address information on a network where the 3D illustration data is stored, or the like.

[0047] The thumbnail generation unit 211 acquires 3D illustration data from the illustration information storage unit 212 when 3D illustration data is added to the illustration information storage unit 212, or at predetermined intervals, and converts the 3D illustration data into 2D thumbnail data. The 2D thumbnail data is, for example, two-dimensional line drawing data as shown in FIG. 6(c), and is line drawing data equivalent to reducing and two-dimensionalizing the 3D illustration data. The 2D thumbnail data corresponds to a two-dimensional image data format (e.g., BMP format). The 2D thumbnail data is associated with identification information of the 3D illustration data. The thumbnail generation unit 211 obtains multiple planar coordinates obtained by projecting multiple spatial coordinates included in the 3D illustration data onto a predetermined plane, and reduces the two-dimensional line drawing data corresponding to the obtained multiple planar coordinates to generate reduced and two-dimensional 2D thumbnail data from the 3D illustration data. The thumbnail generation unit 211 additionally stores the 2D thumbnail data in the thumbnail information storage unit 210 in a form associated with the identification information of the 3D illustration data.

[0048] As a result, thumbnail information 210a as shown in FIG. 8 is stored in the thumbnail information storage unit 210. FIG. 8 is a diagram showing the data structure of thumbnail information 210a stored in the thumbnail information storage unit 210. Thumbnail information 210a is information in which keywords, 3D illustration data, and 2D thumbnail data are associated with one or more pieces of 2D thumbnail data. Thumbnail information 210a includes, as first association information, information in which keywords are associated with 2D thumbnail data. As shown in FIG. 8, multiple keywords may be set for one piece of 2D thumbnail data. Furthermore, association with 2D thumbnail data, which is an example of an image, is not limited to keywords. For example, if recognition unit 103 generates a characteristic waveform pattern contained in an audio signal as a recognition result, it can also be associated with information on the waveform pattern. Furthermore, thumbnail information 210a includes, as second association information, information in which 2D thumbnail data is associated with 3D illustration data. For example, the thumbnail information 210a has a keyword field 210a1, an identification information field 210a2, and an access information field 210a3. The keyword field 210a1 records a keyword for calling up 2D thumbnail data, but is blank if no keyword is registered. The identification information field 210a2 records information for identifying 3D illustration data linked to the 2D thumbnail data, such as an ID number of the 3D illustration data. The access information field 210a3 records information for accessing the 2D thumbnail data, such as the file name of the 2D thumbnail data. The information for accessing the 2D thumbnail data is not limited to a file name and may also be address information on a network where the 2D thumbnail is stored. The keyword may be any of words, phonetic characters, ideograms, sentences, numbers, or a combination thereof.

[0049] When 3D illustration data is added to illustration information storage unit 212, or at predetermined intervals, registered character input unit 216 receives character information to be linked to the 3D illustration data. Registered character input unit 216 accepts the character information in a form associated with identification information of the 3D illustration data. As indicated by the dotted arrow, registered character input unit 216 may acquire, from illustration information storage unit 212, identification information of multiple pieces of 3D illustration data that are candidates to be linked with the character information, and select and input identification information from the multiple pieces of 3D illustration data. Registered character input unit 216 supplies the character information to keyword generation unit 215 in a form associated with the identification information of the 3D illustration data.

[0050] When the keywording unit 215 receives the character information associated with the identification information of the three-dimensional illustration data, it accesses the thumbnail information storage unit 210 and writes the character information into the keyword field 210a1 corresponding to the identification information of the three-dimensional illustration data.

[0051] As a result, in the thumbnail information 210a stored in the thumbnail information storage unit 210, one or more keywords are additionally recorded in the keyword column 210a1, as shown in FIG.

[0052] The data size of 2D thumbnail data is significantly smaller than that of 3D illustration data. By using thumbnails with a small data size when searching for illustrations based on language information, it is possible to minimize the time lag when retrieving images.

[0053] The data storage in the selection tendency storage unit 209 will be described with reference to Fig. 5. The context analysis unit 207 analyzes the context based on the linguistic information recognition results supplied from the linguistic information input unit 100. The selection information in the thumbnail selection unit 204 is stored as the user's selection tendency in the selection tendency storage unit 209. Based on the analysis results of the context analysis unit 207 and the storage results of the selection tendency storage unit 209, the priority determination unit 208 determines the priority for displaying the 2D thumbnail data and supplies the determination result to the thumbnail display unit 203. The thumbnail display unit 203 displays the 2D thumbnail data on the display 506 in descending order of priority according to the determination result of the priority determination unit 208.

[0054] An example of the priority may be a priority determined in accordance with a foreign language as language information. The priority determination unit 208 may determine a priority such that the thumbnails to be displayed change depending on the type of language. For example, when the word "school" is detected in an African language, the priority is determined so that an illustration of a school commonly seen in Africa is displayed preferentially instead of an illustration of a typical school in Japan.

[0055] The illustration display method will be described with reference to FIG. 5. Thumbnail search unit 202, serving as a first image search unit, searches for 2D thumbnail data based on the results of the language information recognition. Thumbnail search unit 202 accesses thumbnail information storage unit 210 and searches thumbnail information 210a for keywords contained in the text information resulting from the language information recognition. Thumbnail search unit 202 retrieves one or more pieces of 2D thumbnail data corresponding to the keywords and identification information for the associated 3D illustration data as search results, and supplies these to thumbnail display unit 203. Thumbnail display unit 203, serving as a first image display unit, supplies the retrieved one or more pieces of 2D thumbnail data to drawing information generation unit 400 (see FIG. 2) via connection unit 3. In response, drawing information generation unit 400 displays the retrieved one or more pieces of 2D thumbnail data in a specific location on display 506 of client terminal 1. In this way, thumbnail display unit 203, serving as a first image display unit, displays an image corresponding to the input language information on a display unit that displays images, based on the results of recognition by recognition unit 103. Thumbnail selection unit 204, serving as a selection means, selects 2D thumbnail data from one or more 2D thumbnail data displayed on display 506 in response to a user selection operation, for example, an operation of selecting 2D thumbnail data displayed on display 506 with a pointing device, and supplies identification information of 3D illustration data associated with the selected 2D thumbnail data to illustration search unit 205, serving as a second image search means. Illustration search unit 205 searches for 3D illustration data based on the identification information of the 3D illustration data. Illustration search unit 205 accesses illustration information storage unit 212, retrieves the 3D illustration data corresponding to the identification information as a search result, and supplies the retrieved 3D illustration data to illustration display unit 206. Illustration display unit 206, serving as a second image display means, supplies the retrieved 3D illustration data to drawing information generation unit 400 (see FIG. 2 ) via connection unit 3. In response to this, drawing information generation unit 400 displays the retrieved 3D illustration data on display 506 of client terminal 1.

[0056] Alternatively, the three-dimensional illustration data may be displayed as thumbnails without being thumbnailized, as will be described in detail below.

[0057] Data storage in the three-dimensional model storage unit 217, which is the first storage means, will be described with reference to FIGS.

[0058] As shown in Fig. 9, when three-dimensional illustration data is displayed as thumbnails, screen display control unit 200 does not need illustration search unit 205, thumbnail generation unit 211, and illustration information storage unit 212 shown in Fig. 5. Also, as shown in Fig. 9, screen display control unit 200 includes a three-dimensional model storage unit 217 instead of thumbnail information storage unit 210, and a three-dimensional model search unit 218 instead of thumbnail search unit 202.

[0059] The three-dimensional model input unit 214 receives three-dimensional model data. The three-dimensional model data is, for example, polygon data as shown in FIG. 6(a) and includes a plurality of spatial coordinates. The three-dimensional model data corresponds to a three-dimensional image data format (for example, OBJ format). The three-dimensional model input unit 214 supplies the three-dimensional model data to the three-dimensional model storage unit 217.

[0060] Character information to be linked to the three-dimensional model data is input to the registration character input unit 216 at the timing when the three-dimensional model data is added to the three-dimensional model accumulation unit 217, or at predetermined intervals. The registration character input unit 216 accepts the character information in a form associated with the identification information of the three-dimensional model data.

[0061] When the keywording unit 215 receives the character information in a form associated with the identification information of the three-dimensional model data, it accesses the three-dimensional model storage unit 217 and writes the character information additionally into the keyword field 217a1 corresponding to the identification information of the three-dimensional model data.

[0062] As a result, the three-dimensional model information 217a shown in FIG. 10 is stored in the three-dimensional model storage unit 217. FIG. 10 is a diagram showing the data structure of the three-dimensional model information 217a stored in the three-dimensional model storage unit 217. The three-dimensional model information 217a includes information in which keywords are associated with three-dimensional model data. As shown in FIG. 10, a plurality of keywords may be set for one piece of three-dimensional model data. Furthermore, association with a three-dimensional model, which is an example of an image, is not limited to a keyword. For example, if the recognition unit 3 generates a characteristic waveform pattern contained in an audio signal as a recognition result, it can also be associated with information on the waveform pattern.

[0063] For example, the three-dimensional model information 217a has a keyword column 217a1, an identification information column 217a2, and an access information column 217a3. The keyword column 217a1 records a keyword for calling up three-dimensional model data, but is blank if no keyword is registered. The identification information column 217a2 records information for identifying the three-dimensional model data, such as an ID number of the three-dimensional model data. The access information column 217a3 records information for accessing the three-dimensional model data, such as a file name of the three-dimensional model data. The information for accessing the three-dimensional model data is not limited to a file name, and may also be an address on a network where the three-dimensional model data is stored, or the like.

[0064] The method for displaying an illustration will now be described. The 3D model search unit 218 searches for 3D model data based on the results of the language information recognition. The 3D model search unit 218 accesses the 3D model storage unit 217 and searches for 3D model information 217a using keywords contained in the character information resulting from the language information recognition. The 3D model search unit 218 retrieves one or more pieces of 3D model data corresponding to the character information (e.g., keywords) as search results and supplies them to the illustration conversion unit 213. The illustration conversion unit 213 converts the 3D model data into 3D illustration data. The 3D illustration data is, for example, three-dimensional line drawing data as shown in FIG. 6(b) and includes multiple spatial coordinates. The 3D illustration data is compatible with a three-dimensional image data format (e.g., OBJ format). The illustration conversion unit 213 extracts line drawing information from the 3D model data by identifying the three-dimensional positions of the edges of the three-dimensional shape represented by the 3D model data and connecting them with line drawings, thereby generating 3D illustration data.

[0065] The thumbnail display unit 203, serving as a first image display unit, supplies the generated 3D illustration data to the drawing information generation unit 400 (see FIG. 2 ) via the connection unit 3. In response, the drawing information generation unit 400 displays the 3D illustration data at a specific location on the display 506 of the client terminal 1. In this way, the thumbnail display unit 203, serving as a first image display unit, displays an image corresponding to the input linguistic information on the display unit for displaying images, based on the recognition result by the recognition unit 103. The first display control unit may directly display the search result image without performing illustration by the illustration generation unit 213. Both the image of the 3D model data and the image of the 3D illustration data resulting from the illustration generation of the 3D model data correspond to the input linguistic information. The thumbnail selection unit 204, serving as a selection unit, selects 3D illustration data from one or more 3D illustration data displayed on the display 506 in response to a user selection operation, for example, an operation of selecting 3D illustration data displayed on the display 506 with a pointing device, and supplies the selected 3D illustration data to the illustration display unit 206. Illustration display unit 206, which serves as second image display means, supplies the retrieved three-dimensional illustration data to drawing information generation unit 400 (see FIG. 2) via connection unit 3. In response to this, drawing information generation unit 400 displays the retrieved three-dimensional illustration data on display 506 of client terminal 1.

[0066] The search target of the three-dimensional model search unit 218 as the first image search means may be any image, such as a still image, a video, an illustration, a photograph, a two-dimensional illustration, a three-dimensional illustration, an animation, a stereoscopic image, or the like.

[0067] The display objects of the thumbnail display unit 203 as the first image display means and the illustration display unit 206 as the second image display means may also be images, and may be any of still images, videos, illustrations, photographs, 2D illustrations, 3D illustrations, animations, stereoscopic images, etc.

[0068] The three-dimensional model accumulation unit 217, which is the first storage means, accumulates three-dimensional model information 217a, but is not limited to this as long as it is image information, and may be any other information such as still images, videos, illustrations, photographs, two-dimensional illustrations, three-dimensional illustrations, animations, stereoscopic images, etc.

[0069] Next, an example in which the priority image presenting unit 220 presents priority images based on predictions made by machine learning will be described.

[0070] <Function block> Fig. 11 is a functional block diagram of the priority image presenting unit 220 in the embodiment. As shown in Fig. 11, the priority image presenting unit 220 can include a selected image acquiring unit 221, a training data storage unit 222, a machine learning unit 223, a trained model storage unit 224, an inference unit 225, and a notification unit 226. Each of these will be described below.

[0071] The selected image acquisition unit 221 acquires text information and selected image information, such as thumbnail images, from the thumbnail selection unit 204 .

[0072] <Learning Phase> The teacher data storage unit 222 stores teacher data for machine learning. The teacher data in the teacher data storage unit 222 includes character information acquired and accumulated over a certain period of time by the selected image acquisition unit 221, and data indicating the selected image, such as an image file name. The teacher data storage unit 222 stores teacher data including a plurality of data pairs each consisting of character information and an image file name.

[0073] The machine learning unit 223 generates a trained model for deriving image information to be selected next from the received selected image data. Specifically, the machine learning unit 223 performs machine learning using training data in which the received character information is used as input data and the image selected after the character information is received is used as output data, thereby generating a trained model. The machine learning unit 223 also stores the generated trained model in the trained model storage unit 224.

[0074] The trained model storage unit 224 stores trained models generated by the machine learning unit 223.

[0075] <Inference phase> The inference unit 225 obtains the currently selected image and infers the next selected image.

[0076] Specifically, the inference unit 225 acquires character information and information for identifying the selected image (for example, an image file name) from the selected image acquisition unit 221. The inference unit 225 also inputs the character information to the trained model in the trained model storage unit 224, causing it to output image information (for example, an image file name) to be selected next.

[0077] The notification unit 226 notifies the thumbnail display unit 203 of the image information (for example, image file name) to be selected next, which is output from the inference unit 225, as the image to be preferentially displayed.

[0078] In the above-mentioned <learning phase>, the machine learning unit 223 may acquire and learn user information such as a login ID for the user's communication system 4 in addition to the character information. In this case, the machine learning unit 223 can further learn an image corresponding to the user who inputted the language information.

[0079] The drawing operation unit 300 shown in Figure 2 can be functionally configured as shown in Figure 12. Figure 12 is a diagram showing the configuration of the drawing operation unit 300.

[0080] The drawing operation unit 300 includes an illustration editing unit 301 , a fixing unit 302 , a drawing unit 303 , a two-dimensional data editing input unit 304 , and an output unit 305 .

[0081] Illustration editing unit 301, serving as a first editing means (3D illustration data editing input means), receives 3D illustration data from screen display control unit 200 and performs 3D editing processing on the 3D illustration data. During the 3D editing processing, illustration editing unit 301 accepts 3D rotation operations, 3D movement operations, 3D enlargement operations, 3D reduction operations, and other operations performed by the user, and supplies these operation requests to drawing information generation unit 400 (see FIG. 2 ) via drawing unit 303, output unit 305, and connection unit 3. A 3D rotation operation is an operation in which multiple spatial coordinates included in the 3D illustration data are changed so that they are rotated three-dimensionally around a predetermined axis while maintaining their relative positional relationships. A 3D enlargement operation is an operation in which multiple spatial coordinates included in the 3D illustration data are changed so that they move radially away from a predetermined point at an equal distance rate. A 3D reduction operation is an operation in which multiple spatial coordinates included in the 3D illustration data are changed so that they move radially closer to a predetermined point at an equal distance rate. In response to these operation requests, the drawing information generation unit 400 changes the display format of the 3D illustration data in the drawing unit 303 of the client terminal 1. This allows the position, size, and orientation of the 3D illustration data on the display 506 of the client terminal 1 to be changed three-dimensionally.

[0082] Fixing unit 302, which serves as a conversion means, converts (fixes) 3D illustration data that can be edited in a 3D manner into 2D illustration data that can be edited in a 2D manner in response to a predetermined operation (e.g., an operation that activates 2D editing, more specifically, an operation that transitions to a state where 2D editing is possible). The 2D illustration data is 2D line drawing data, such as that shown in FIG. 6(d), and is line drawing data that corresponds to 3D illustration data that has been edited in a 3D manner and converted into 2D. The 2D illustration data corresponds to a 2D image data format (e.g., BMP format). Fixing unit 302 converts the 3D illustration data into 2D illustration data by, for example, determining a plurality of plane coordinates obtained by projecting a plurality of spatial coordinates included in the 3D illustration data onto a plane corresponding to the screen of display 506, and generating 2D line drawing data corresponding to the determined plurality of plane coordinates, thereby converting the 3D illustration data into 2D illustration data and fixing it in drawing unit 303.

[0083] The 2D data edit input unit 304, which serves as a second editing means (2D data edit input means), performs two-dimensional editing processing on the 2D illustration data. In the two-dimensional editing processing, the 2D data edit input unit 304 accepts user operations such as two-dimensional rotation, two-dimensional movement, two-dimensional enlargement, two-dimensional reduction, erasure, adding handwritten lines, and coloring, and supplies these operation requests to the drawing information generation unit 400 (see FIG. 2 ) via the drawing unit 303, output unit 305, and connection unit 3. The two-dimensional rotation operation is an operation in which multiple planar coordinates included in the 2D illustration data are changed so that they are rotated two-dimensionally around a specified point while maintaining their relative positional relationship. The two-dimensional enlargement operation is an operation in which multiple planar coordinates included in the 2D illustration data are changed so that they move radially away from a specified point at equal distances. A two-dimensional reduction operation is an operation in which multiple planar coordinates included in the two-dimensional illustration data are changed so that they approach a predetermined point radially at equal distances. A partial erasure operation is an operation in which a portion of multiple planar coordinates included in the two-dimensional illustration data is deleted. A handwritten line drawing operation is an operation in which multiple planar coordinates corresponding to the handwritten line drawing are added to multiple planar coordinates included in the two-dimensional illustration data. A coloring operation is an operation in which multiple planar coordinates associated with predetermined color attributes are added to multiple planar coordinates included in the two-dimensional illustration data. In response to these operation requests, the drawing information generation unit 400 changes the display format of the two-dimensional illustration data in the drawing unit 303 of the client terminal 1. As a result, the position, size, and orientation of the two-dimensional illustration data on the display 506 of the client terminal 1 are changed two-dimensionally, a portion of the two-dimensional illustration data is erased, handwritten line drawings are added to the two-dimensional illustration data, or color is added to the two-dimensional illustration data.

[0084] Input to the drawing unit 303 is also possible using a mouse, a finger, a stylus pen, gestures, etc. Information from the drawing unit 303 is output by the output unit 305 to the drawing information generation unit 400. In response to this, the drawing information generation unit 400 generates drawing information to which the handwritten line drawing, etc. input in the drawing unit 303 of the client terminal 1 is added. As a result, the handwritten line drawing, etc. is displayed on the display 506 of the client terminal 1.

[0085] These units are functions or means that are realized by operating in accordance with instructions from the CPU 501 in accordance with a program loaded from the HD 504 onto the RAM 503 .

[0086] Next, the operation of the visual communication system 4 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the operation of the visual communication system.

[0087] The visual communication system 4 performs a registration process (S1) in which predetermined information is registered in preparation for assisting communication. Thereafter, the visual communication system 4 waits until a startup request is received from the user (No in S2). When a startup request is received from the user (Yes in S2), the visual communication system 4 starts the program 500a and starts the communication assistance process (S3). The visual communication system 4 continues to perform the communication assistance process (S3) until a termination request is received from the user (No in S4). When a termination request is received from the user (Yes in S4), the visual communication system 4 terminates the process.

[0088] Next, details of the registration process (S1) will be explained using Fig. 14. Fig. 14 is a flowchart showing the flow of the registration process.

[0089] When program 500a is started, visual communication system 4 waits until a request to register three-dimensional model data is received (No in S11). When a request to register three-dimensional model data is received (Yes in S11), visual communication system 4 receives the three-dimensional model data (S12). The three-dimensional model data is, for example, polygon data (see FIG. 6(a)). Visual communication system 4 converts the three-dimensional model data into three-dimensional illustration data (S13). That is, visual communication system 4 converts the three-dimensional model data into three-dimensional illustration data. The three-dimensional illustration data is, for example, three-dimensional line drawing data (see FIG. 6(b)). Visual communication system 4 additionally stores the three-dimensional illustration data in illustration information storage unit 212. As a result, illustration information 212a (see FIG. 7) is registered in illustration information storage unit 212.

[0090] The visual communication system 4 thumbnails the 3D illustration data (S14). That is, the visual communication system 4 converts the 3D illustration data into 2D thumbnail data. The 2D thumbnail data is, for example, two-dimensional line drawing data (see FIG. 6(c)). The visual communication system 4 additionally stores the 2D thumbnail data in the thumbnail information storage unit 210 in association with identification information of the 3D illustration data. As a result, the 2D thumbnail data and the identification information of the 3D illustration data in thumbnail information 210a (see FIG. 8) are registered in the thumbnail information storage unit 210.

[0091] The visual communication system 4 then waits until a keyword registration request is received (No in S15). When a keyword registration request is received (Yes in S15), the visual communication system 4 inputs character information associated with the identification information of the 3D illustration data (S16). The visual communication system 4 converts the character information into a keyword (S17). That is, the visual communication system 4 accesses the thumbnail information storage unit 210 and additionally writes the character information in the keyword field 210a1 corresponding to the identification information of the 3D illustration data. This registers the character information (keyword) in the thumbnail information 210a (see FIG. 8) in the thumbnail information storage unit 210. Note that it is not necessary to perform both the registration process and the communication support process as shown in FIG. 13; it is also possible to execute only the communication support process based on the information registered in the thumbnail information storage unit 210 and the illustration information storage unit 212 at that time.

[0092] The flowchart in Fig. 14 has been explained assuming the configuration in Fig. 5, but it can also be applied as appropriate to the configuration in Fig. 9. That is, when applied to the configuration in Fig. 9, steps S13 and S14 are not executed, and instead of step S16, a process is executed in which the character information is associated with the three-dimensional model data, and as a result, three-dimensional model information 217a such as that shown in Fig. 10 is accumulated in three-dimensional model accumulation unit 217.

[0093] Next, the communication support process (S3) will be described in detail with reference to Fig. 15 and Fig. 16. Fig. 15 and Fig. 16 are flowcharts showing the flow of the communication support process. The process of Fig. 15 and the process of Fig. 16 can be performed in parallel with each other.

[0094] In the processing of FIG. 15, when the program 500a is started and an initial screen is displayed on the display 506, the visual communication system 4 waits until a request to turn on the language information input function is received via the initial screen (No in S21). When a request to turn on the language information input function is received (Yes in S21), the visual communication system 4 waits until language information is input (No in S22). When language information is input (Yes in S22), the visual communication system 4 determines whether the language information is voice information (S23). If the language information is voice information (Yes in S23), the visual communication system 4 performs voice recognition processing on the voice information to convert it into character information (S24), and uses the character information as the recognition result. If the language information is not voice information (No in S23), the visual communication system 4 determines whether the language information is a handwritten character image (S25). If the language information is a handwritten character image (Yes in S25), the visual communication system 4 performs text recognition processing on the handwritten character image to convert it into character information (S26), and sets the character information as the recognition result. If the language information is not a handwritten character image, that is, if the language information is character information typed on the keyboard 511 (No in S25), the visual communication system 4 sets the character information as the recognition result, and proceeds to S27. Steps S23 to S26 are executed by the recognition unit 103 in the visual communication system 4, for example.

[0095] The visual communication system 4 accesses the thumbnail information storage unit 210 and searches for 2D thumbnail data using keywords contained in the text information that is the result of the language information recognition (S27).The visual communication system 4 displays one or more pieces of searched 2D thumbnail data in a specific location on the display 506 (S28).

[0096] The visual communication system 4 can rapidly repeat the processes of S22 to S28 until a request to turn off the language information input function is received (No in S29). That is, the visual communication system 4 can rapidly update the display of new thumbnail images on the display screen every time language information is received from the user. This allows the user to select one or more thumbnail images that closely resemble the scene in their mind when they spoke or entered text, without interrupting the communication of information through conversation or the like.

[0097] When a request to turn off the language information input function is received (Yes in S29), the visual communication system 4 ends the processing of FIG.

[0098] The flowchart in Fig. 15 has been explained assuming the configuration in Fig. 5, but it can also be applied as appropriate to the configuration in Fig. 9. That is, when applied to the configuration in Fig. 9, the three-dimensional model data is searched for in step S27, and instead of step S28, the three-dimensional model data is illustrated and three-dimensional illustration data is displayed.

[0099] 16, visual communication system 4 waits until 2D thumbnail data is selected from the one or more pieces of 2D thumbnail data displayed in S28 (No in S31). When 2D thumbnail data is selected from the one or more pieces of 2D thumbnail data displayed in S28 (Yes in S31), visual communication system 4 accesses illustration information storage unit 212 and searches for 3D illustration data associated with the selected 2D thumbnail data (S32). Visual communication system 4 displays the searched 3D illustration data on display 506 (S33).

[0100] The visual communication system 4 waits until an editing operation is performed on the three-dimensional illustration data (No in S34), and when an editing operation is performed on the three-dimensional illustration data (Yes in S34), it performs a first editing process (S35). The first editing process is a three-dimensional editing process. In the first editing process, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, and the like. In response to these operation requests, the visual communication system 4 changes the display form of the three-dimensional illustration data displayed on the display 506 (S36). The visual communication system 4 repeats the processes of S34 to S36 until the three-dimensional illustration data is fixed to two-dimensional illustration data (No in S37).

[0101] When the 3D illustration data is fixed to 2D illustration data (Yes in S37), the visual communication system 4 waits until an editing operation is performed on the 2D illustration data (No in S38). When an editing operation is performed on the 2D illustration data (Yes in S38), the visual communication system 4 performs a second editing process (S39). The second editing process is a two-dimensional editing process. In the second editing process, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add handwritten line drawings, coloring operations, and the like. In response to these operation requests, the visual communication system 4 changes the display format of the 2D illustration data displayed on the display 506 (S40). The visual communication system 4 repeats the processes of S38 to S40 until a request to complete editing is received (No in S41). That is, by performing the first stage of editing processing in a loop of S34 to S37 and the second stage of editing processing in a loop of S38 to S41, an illustration that closely resembles the scene that is in the mind when the speech or character input is made can be accurately expressed in a short time.

[0102] When a request to complete editing is received (Yes in S41), the visual communication system 4 ends the processing of FIG.

[0103] The flowchart in Fig. 16 has been explained assuming the configuration in Fig. 5, but it can also be applied to the configuration in Fig. 9 as appropriate. That is, when applied to the configuration in Fig. 9, the 3D illustration data is displayed when the processing in Fig. 15 is completed, so in step S31, the process waits until 3D illustration data is selected instead of 2D thumbnail data. If selected, step S32 is not executed, and the selected 3D illustration data is displayed in step S33, and the processing from step S34 onwards is executed.

[0104] Next, the configuration of the screen displayed on the display 506 of the client terminal 1 by the visual communication system 4 (the display screen by the visual communication system 4) will be described with reference to Fig. 17. Fig. 17 is a diagram showing the configuration of the display screen by the visual communication system 4.

[0105] As shown in FIG. 17, the display screen of the visual communication system 4 includes a drawing result display area 600, an operation palette 700, and a thumbnail display area 800.

[0106] The drawing result display area 600 is an area where the drawing information generation result by the drawing information generation unit 400 (see FIG. 2) is output. The operation palette 700 has functions arranged therein that are used to call up each input unit, such as the voice input unit 101, the character input unit 102 (see FIG. 4), and the 2D data edit input unit 304 (see FIG. 12). The thumbnail display area 800 displays the language information recognized by the recognition unit 103 (see FIG. 4) and the 2D thumbnail data output by the thumbnail display unit 203 (see FIG. 5).

[0107] The layout and design shape of each area shown in Figure 17 are merely examples and do not limit the scope of rights. For example, eight thumbnails surrounded by circular frames are displayed in 800, but there are no restrictions on the presence or absence of frames, their shape, or the number of thumbnails displayed. Also, recognized language information does not need to be displayed.

[0108] Next, the general operation of the screen (display screen by the visual communication system 4) displayed on the display 506 of the client terminal 1 by the visual communication system 4 will be described with reference to Fig. 18. Fig. 18 is a diagram showing the general operation of the display screen by the visual communication system 4. Fig. 18 shows the operation of calling up 2D thumbnail data based on language information.

[0109] 18(a), the visual communication system 4 transitions to a language information recognition mode. In the language information recognition mode, when the visual communication system 4 acquires language information (first language information) by speech, handwriting, typing, or the like, the language information recognition result and one or more pieces of 2D thumbnail data linked to the recognition result are displayed in the thumbnail display area 800. Furthermore, when the visual communication system 4 detects a press of a 2D data edit button 702, the system transitions to a drawing result edit mode in which the drawing result display area 600 in FIG. 17 can be edited.

[0110] For example, in Figure 18(a), the visual communication system 4 recognizes handwritten characters drawn on the drawing result display area 600, and displays the character information of the recognition result and one or more 2D thumbnail data associated with it in the thumbnail display area 800.

[0111] When the visual communication system 4 further acquires language information (second language information) by speaking, handwriting, typing, etc., at least a portion of the one or more two-dimensional thumbnail data displayed in the thumbnail display area 800 is changed, and the language information recognition result and the one or more two-dimensional thumbnail data linked to the recognition result are updated and displayed.

[0112] For example, when the visual communication system 4 recognizes an utterance such as that shown in Fig. 18(b) in the state shown in Fig. 18(a), the visual communication system 4 changes the state of the thumbnail display area 800 in real time according to the character information of the recognition result. Fig. 18(b) shows the state in which the visual communication system 4 updates and displays the new character information of the recognition result in the thumbnail display area 800 based on one or more pieces of 2D thumbnail data associated with the new character information.

[0113] Specifically, at least a portion of the thumbnail images already displayed is changed and displayed on the display (display means) 506 in accordance with the thumbnail images corresponding to the newly input language information (second language information).

[0114] Changing at least a part of the display format of a thumbnail image that is already displayed means, for example, changing the position or size of a thumbnail image that is already displayed in order to add a thumbnail image corresponding to newly input language information to the thumbnail image that is already displayed, deleting a thumbnail image that corresponds to newly input language information in order to display it in place of the image that is already displayed, or changing the image surrounding the thumbnail image that is already displayed.

[0115] The visual communication system 4 may change the display in the thumbnail display area 800 not only when speech is received but also whenever new handwritten or typed characters are recognized.

[0116] Furthermore, if the 2D thumbnail data cannot all be displayed in the thumbnail display area 800, the visual communication system 4 may display the data in the thumbnail display area 800 in order from oldest to newest retrieved 2D thumbnail data.

[0117] In FIG. 18, the switch to switch to the language information recognition mode is the language information recognition button 701, but the method is not limited to this and may be, for example, the utterance of a specific word or the input of a command instead of a button.

[0118] 18, the visual communication system 4 can quickly update the thumbnail images on the display screen every time it receives linguistic information from the user. This allows the user to select one or more thumbnail images that closely resemble the scene in their mind when they speak or input text, without interrupting the communication of information through conversation or the like.

[0119] Next, the flow of the two-stage editing process of an illustration by the visual communication system 4 will be described with reference to Fig. 19. Fig. 19 is a diagram showing the flow of the two-stage editing process.

[0120] In FIG. 19(a), the visual communication system 4 displays the character information LI of "person" and the two-dimensional thumbnail data SM1 to SM6 linked thereto in the thumbnail display area 800 in accordance with the linguistic information "person."

[0121] The character information LI is displayed in a text data area 801. In Fig. 19(a), as an example, a circular dotted line is displayed so as to be visible to the user, and the character information LI is displayed inside it.

[0122] The two-dimensional thumbnail data SM1 to SM6 are displayed in a circumferential arrangement in the thumbnail data area 802 so as not to overlap one another. This is not a limitation, and the thumbnail data may be displayed overlapping one another or randomly. In particular, in this example, the data is displayed in the area between the two dashed-dotted lines. To make it easier for the user to recognize the thumbnail data area 802, for example, a line visible to the user may actually be displayed at the dashed-dotted line position.

[0123] When the 2D thumbnail data SM5 is selected, the visual communication system 4 calls up the 3D illustration data linked to the 2D thumbnail data SM5 and displays it in the drawing result display area 600, as shown in FIG. 19(b).

[0124] In the three-dimensional editing process, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, etc., and changes the display form of the three-dimensional illustration data three-dimensionally in accordance with these operation requests, as shown in Figure 19(c).

[0125] Upon receiving a predetermined trigger operation (for example, pressing a button for a two-dimensional editing operation (for example, the two-dimensional data editing button 702 shown in Figure 18(a))), the visual communication system 4 fixes the three-dimensional illustration data to two-dimensional illustration data, as shown in Figure 19(d).

[0126] In two-dimensional editing processing, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn line drawings, coloring operations, etc., and changes the display form of the two-dimensional illustration data two-dimensionally in accordance with these operation requests, as shown in Figure 19(e).

[0127] As illustrated in Figure 19, the first stage of editing processing (Figure 19(b) and Figure 19(c)) and the second stage of editing processing (Figure 19(e)) make it possible to accurately express, in a short amount of time, an illustration that closely resembles the scene that occurs in the mind when speech or text is input.

[0128] Next, detailed operations of the screens (display screens by the visual communication system 4) displayed on the display 506 of the client terminal 1 by the visual communication system 4 will be described with reference to Figs. 20 to 27. Figs. 20 to 27 are diagrams each showing detailed operations of the display screens by the visual communication system 4. Fig. 18 shows operations in which 2D thumbnail data is called based on language information, a thumbnail is selected, and illustration editing is performed.

[0129] When it is detected that the voice input button 7011 in the operation palette 700 shown in Fig. 20(a) has been pressed, the visual communication system 4 turns on the language information recognition function and displays an icon 8021 indicating that voice input is in a standby state in the thumbnail display area 800, as shown in Fig. 20(b). When the key input button 7012 in the operation palette 700 is pressed, typing input via the keyboard becomes possible, and an icon indicating that typing input is in a standby state is displayed. These icons 8021 and the icon indicating that typing input is in a standby state are examples of language information recognition state display icons.

[0130] When it is detected that "people and vehicles" has been uttered, the visual communication system 4 displays the text information LI1 of "people and vehicles" and the two-dimensional thumbnail data SM11 to SM18 linked to it in the thumbnail display area 800, as shown in Fig. 21. Note that the thumbnail data linked to the text information may be thumbnail data linked to the entire text information of "people and vehicles," or may be a group of thumbnail data broken down into keywords such as "people" and "vehicles" and linked to each keyword.

[0131] Furthermore, when it detects that "animals" have been uttered, the visual communication system 4 updates and displays the text information LI2 of "people and vehicle animals" and the 2D thumbnail data SM21 to SM28 linked to it in the thumbnail display area 800, as shown in Fig. 22. That is, every time the visual communication system 4 detects the input of linguistic information, it updates and displays the 2D thumbnail data in real time. Note that the thumbnail data linked to the text information may be thumbnail data linked to the entire text information of "people and vehicle animals," or it may be a group of thumbnail data broken down into keywords such as "people," "vehicles," and "animals" and linked to each keyword.

[0132] 21 and 22, the visual communication system 4 can quickly update the thumbnail images on the display screen every time it receives linguistic information from the user. This allows the user to select one or more thumbnail images that closely resemble the scene in their mind when they spoke or entered text, without interrupting the communication of information through conversation or the like.

[0133] When it is detected that the 2D thumbnail data SM14 has been selected on the screen of Figure 21, the visual communication system 4 calls up the 3D illustration data IL1 linked to the 2D thumbnail data SM14 and displays it in the drawing result display area 600, as shown in Figure 23.

[0134] In the three-dimensional editing process, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, etc., and changes the display form of the three-dimensional illustration data IL1 three-dimensionally in accordance with these operation requests, as shown in Figure 24-1.

[0135] As shown in FIGS. 24-2 and 24-3, the user can drag the text displayed in the text data area 801 and display it in the drawing result display area 600 as a text image.

[0136] In FIG. 24-2, for example, the user selects a portion of the text displayed in the text data area 801 by dragging it with the cursor. When the visual communication system 4 detects that the text has been selected and dragged, it converts the text data into image data. Then, as shown in FIG. 24-3, the converted text image data is displayed in the drawing result display area 600. The converted text image data may be either a 2D image or a 3D image, and can be moved, enlarged, reduced, rotated, and various edits can be performed using the 2D data edit button 702 within the drawing result display area 600.

[0137] When it is detected that any of the buttons for two-dimensional editing (the retouch button 7013, the color button 7014, or the eraser button 7015) has been pressed, the visual communication system 4 fixes the three-dimensional illustration data IL1 to two-dimensional illustration data IL2, as shown in Fig. 25. In this embodiment, as an example, the system determines that the pressing of this button for two-dimensional editing is a request to turn off the language recognition function, and transitions to a state where two-dimensional editing is possible.

[0138] In the two-dimensional editing process, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn line drawings, coloring operations, etc., and changes the display form of the two-dimensional illustration data IL2 two-dimensionally in accordance with these operation requests, as shown in Figures 25, 26, and 27.

[0139] In FIG. 25, the retouch button 7013 is used to add a line drawing of a landscape that will be the background of the two-dimensional illustration data IL2 by hand, and the color button 7014 is used to color the two-dimensional illustration data IL2.

[0140] 25, the language information recognition status display icon, character information, and thumbnail data that were displayed in the thumbnail display area 800 in Fig. 23 and Fig. 24-1 are hidden. The timing for hiding the various pieces of information in the thumbnail display area 800 can be selected as appropriate, such as when it is detected that one of the buttons for two-dimensional editing (the retouch button 7013, the color button 7014, or the eraser button 7015) has been pressed, when fixation is executed, or when two-dimensional editing processing is started by the user.

[0141] Furthermore, even after two-dimensional editing has been performed, by pressing the voice input button 7011 or the key input button 7012, the language information recognition function can be turned on, various information can be displayed in the thumbnail display area 800, new three-dimensional illustration data can be displayed and edited on the image that has been two-dimensionally edited, and new fixed two-dimensional illustration data can be added.

[0142] In FIG. 26, eraser button 7015 is used to delete part of the line drawing in the two-dimensional illustration data IL2'.

[0143] In FIG. 27, a line drawing is added to the two-dimensional illustration data IL2'' by pressing the add button 7013.

[0144] As illustrated in Figures 23 to 27, the first stage editing process (Figures 23, 24-1) and the second stage editing process (Figures 25 to 27) allow for accurate, in a short amount of time, illustrations that closely resemble the scene that occurs in the mind when speech or text is input.

[0145] 17 to 27 have been described assuming the configuration of FIG. 5, but can also be applied to the configuration of FIG. 9 as appropriate. That is, in the configuration of FIG. 9, visual communication system 4 searches for a 3D model corresponding to linguistic information and illustrates the 3D model corresponding to the linguistic information. The 3D illustration data resulting from this illustration can be displayed in thumbnail data area 802 in FIGS. 17 to 27. Then, when visual communication system 4 receives a selection operation to select one of the 3D illustration data displayed in thumbnail data area 802, it displays the selected 3D illustration data in drawing result display area 600.

[0146] As described above, in this embodiment, the visual communication system 4 quickly switches and displays one or more thumbnail images that are candidates for illustrations corresponding to linguistic information in response to changes in the linguistic information. This enables quick selection of an illustration that corresponds to the current linguistic information, and supports fast information transmission in communication situations.

[0147] The concept of the visual communication system 4 may be applied to an online conference system 24. The online conference system 24 may be configured as shown in Fig. 28. Fig. 28 is a system configuration diagram showing the system configuration of the online conference system 24 according to a modified example of the embodiment, and shows an example in which an online conference tool is used as the drawing information generation unit 400.

[0148] The online conference system 24 includes a plurality of client terminals 21a and 21b, a communication management server 22, and a connection unit 23. The connection unit 23 connects the plurality of client terminals 21a and 21b and the communication management server 22 so that they can communicate with each other. The plurality of client terminals 21a and 21b can share a display screen realized by a drawing information generation unit 400.

[0149] 29 is a block diagram showing the functional configuration of the online conference system 24. Each of the client terminals 21a and 21b has a language information input unit 100a and 100b, a screen display control unit 200a and 200b, and a drawing operation unit 300a and 300b. The functions and operations of the language information input unit 100a and 100b, the screen display control unit 200a and 200b, and the drawing operation unit 300a and 300b are similar to the functions and operations of the language information input unit 100, the screen display control unit 200, and the drawing operation unit 300 in the embodiment, respectively.

[0150] The communication management server 22 has a communication management unit 900 in addition to a drawing information generation unit 400 and a storage unit 500. The functions and operations of the drawing information generation unit 400 and the storage unit 500 are similar to those of the drawing information generation unit 400 and the storage unit 500 in the embodiment, respectively.

[0151] The communication management unit 900 transmits and manages audio and camera images received from each of the multiple client terminals 21a and 21b who are conference participants to other client terminals. The drawing information generation unit 400 draws and displays the drawn image by superimposing it on the camera image of the online conference. The communication management unit 900 transmits the camera image with the drawn image superimposed to other client terminals.

[0152] The drawn image may be superimposed on a part of the image or on the entire camera image. In addition to the drawn image, 700, 800, etc. may also be superimposed on the camera image at the same time.

[0153] Here, an example of a usage scenario in which the online conference system 24 described with reference to FIG. 28 is applied to a telepresence robot will be described.

[0154] Fig. 30 is a diagram showing an example of a usage scenario in which the online conference system 24 is applied to a telepresence robot. Fig. 30 shows an example of remote communication between a telepresence robot (robot 10A) installed in base A and an administrator terminal 50 used by an administrator located in a remote location.

[0155] Base A shown in FIG. 30 is, for example, a factory or warehouse where predetermined work is performed by multiple workers (workers A, B, C, and D). As shown in the figure, each of workers A, B, C, and D performs work on a workbench. Furthermore, base A is equipped with multiple robots 10A (robots 10A-1 and 10A-2) that autonomously travel within base A. Furthermore, a manager at a remote location uses a manager terminal 50 to remotely communicate with the robot 10A installed at base A, thereby performing maintenance management of base A, etc.

[0156] Furthermore, multiple notification buttons 20A (20A-1 to 20A-5) are installed within base A. Of these, notification buttons 20A-1 to 20A-3 are installed on the workbench, and notification buttons 20A-4 and 20A-5 are mounted on robots 10A-1 and 10A-2, respectively. Notification button 20A is an operation means for a worker at base A to call a manager at a remote location. Note that the configuration of the operation means is not limited to notification button 20, and may be any device that has a function for calling a manager. For example, if an abnormality occurs during work and the worker wishes to communicate with the manager, the worker can call the manager at a remote location by pressing notification button 20A installed nearby.

[0157] Furthermore, the manager at a remote location receives a notification indicating a call from the user, triggered by the selection of the notification button 20A. Then, the manager can communicate with the worker by starting remote communication between the manager terminal 50 and the robot 10A installed in the site A.

[0158] Furthermore, the robots 10A-1 and 10A-2 that are in remote communication with the manager terminal 50 move to a destination that is set according to the installation position of the notification button 20A pressed by the worker, and perform remote communication with the manager terminal 50. Therefore, the worker who pressed the notification button 20A can use the robot 10A to perform remote communication with the manager at a location where the worker wants to communicate.

[0159] When considering the use of telepresence robots at a site such as base A shown in Figure 30, the administrator terminal 50 and each robot 10A can be applied to the client terminals 21a and 21b in Figure 28 to form the communication system 24 of the present invention.

[0160] Telepresence robots are devices used in various remote locations under different circumstances. Even if a telepresence robot is operated by a user who is unfamiliar with the robot, it can still be used to communicate effectively in remote locations by simply inputting voice or text to obtain images suitable for explaining the situation.

[0161] Furthermore, since keywords can be linked to images, by linking images in advance to the language and technical terms used in a specific remote location where the telepresence robot is used, it is possible to communicate using a common image even with people in the remote location who speak a different language or do not understand the technical terms.

[0162] For example, as shown in FIGS. 31 to 33, a camera image showing a user and a drawing made by the user's gestures acquired by a motion acquisition device can be superimposed on the camera image showing the user. FIG. 31 is a diagram showing a camera image captured by the camera 520 of the client terminal 21a in a modified example of the embodiment, illustrating an example of a camera image showing the user of the client terminal 21a. FIG. 32 is a diagram showing an image superimposed on the camera image of FIG. 31 by the drawing information generating unit 400 in a modified example of the embodiment. FIG. 32 shows an example in which, of the drawing result display area 600, the operation palette 700, and the thumbnail display area 800, the operation palette 700 is not superimposed, but only the drawing result display area 600 and the thumbnail display area 800 are superimposed. FIG. 33 is a diagram showing an image displayed on another client terminal 21b in a modified example of the embodiment. In this case, handwriting made by a user on the display 506 of one client terminal with the user's fingertip is displayed on the other client terminal as an image in which a drawing made by the user's fingertip is superimposed on the camera image.

[0163] Also, a specific gesture may be converted into an illustration and displayed in the drawing result display area 600. For example, if a gesture of pointing up, down, left, or right with a finger is recognized in the video captured by the camera, illustrations of an up arrow, a down arrow, a left arrow, and a right arrow are drawn in the drawing result display area 600. Also, if a clapping gesture is recognized, an illustration of clapping hands is drawn in the drawing result display area 600, or if the trajectory of the fingertip draws a mark such as a circle, a corresponding illustration is drawn, etc. The illustrations drawn in this way can also be edited in various ways using the 2D data edit button 702.

[0164] Furthermore, the operation may be changed depending on the combination of a specific gesture and an area within the drawing display. That is, for example, when the image of a finger shown in Fig. 31 moves from the thumbnail display area 800 within the drawing shown in Fig. 32 to the drawing result display area 600, the gesture is recognized as indicating a drop position for drag and drop, and an operation is performed. On the other hand, when the image of the finger in Fig. 31 is recognized for the first time in the drawing result display area 600, an illustration of an up arrow is drawn in the drawing result display area 600, as shown in Fig. 34. Also, when a gesture of pinching and releasing fingers is recognized around an illustration already drawn in the drawing result display area 600, the illustration moves, and various other combinations are conceivable.

[0165] Another application of image recognition is to perform image recognition on video captured by a camera on the client device of the viewing audience. For example, facial expressions such as smiling, sad, and serious faces can be recognized, and illustrations corresponding to those expressions can be sent to another specific device, such as the client device giving the presentation, to inform the presenter of the audience's reactions. Furthermore, by superimposing illustrations corresponding to facial expressions on drawings and sharing them with the presenter and all other users, all participants can share each other's reactions and the audience's overall reaction to the presentation.

[0166] In this way, visual communication using illustrations through gesture operations can be achieved while transmitting information through conversation, etc.

[0167] In the above-described embodiments, the visual communication system 4 and the online conference system 24 are examples of an image editing system or an image display system, and the client terminal 1, the client terminal 21a, the client terminal 21b, the server 2, and the communication management server 22 are examples of an image editing device or a display device.

[0168] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each of the above-described functions.

[0169] Furthermore, the correspondence table (table) in the specification may be generated by the learning effect of machine learning. Here, machine learning refers to a technology that allows a computer to acquire human-like learning capabilities, in which the computer autonomously generates algorithms necessary for judgments such as data classification from previously acquired learning data, and applies these algorithms to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. The learning method for machine learning is not limited.

[0170] The program 500a executed by the visual communication system 4 or the online conference system 24 may be provided by being pre-installed in a ROM or the like. Alternatively, the program 500a may be provided by being recorded in an installable or executable file on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD). Alternatively, the program 500a may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program 500a may also be provided or distributed via a network such as the Internet. [Explanation of symbols]

[0171] 1. Client terminal 2 Server 3 Connection 4 Visual Communication System 21a, 21b Client terminal 22 Communication Management Server 23 Connection 24 Online conference system 50 Administrator terminal 100 Language information input means 103 Recognition means 202,205,218 Image search methods 203,206 Image display means 204 Selection Method 210,212,217 Storage means 220 Priority Image Presentation Method 506 Display means [Prior art documents] [Patent documents]

[0172] [Patent Document 1] Patent No. 6339529

Claims

1. a language information input means for receiving input of language information; recognition means for recognizing input linguistic information as character information; an image display means for displaying an image corresponding to a keyword included in the character information on a display means for displaying an image based on a recognition result by the recognition means; a selection means for selecting the displayed image; another image display means for displaying an image corresponding to the selected image on the display means in a state in which an editing process operation by a user can be accepted; A communication system comprising:

2. the image display means changes at least a part of the display form of the image that is already being displayed, and displays on the display means an image corresponding to the keyword included in the character information recognized by the recognition means based on the newly input linguistic information. The communication system of claim 1 .

3. the selected image is a thumbnail image of the corresponding image; The communication system of claim 1 .

4. a priority image presenting means for presenting an image to be displayed preferentially from among a plurality of images corresponding to a keyword included in the character information; The priority image presenting means infers the next image to be selected based on a trained model obtained by machine learning using the character information and the image selected by the selection means from among the images displayed in association with the character information as training data, and notifies the image displaying means of the inference. The communication system of claim 1 .

5. a storage means for storing the keywords and the images in association with each other; an image search means for searching for keywords included in the character information; and the display means displays the images associated with the keywords searched by the image search means. A communication system according to any one of claims 1 to 4.

6. the display means is provided in a telepresence robot that is installed in a predetermined base and receives control via remote communication from an administrator terminal; A communication system according to any one of claims 1 to 5.

7. the display means is provided in an administrator terminal capable of remotely controlling the telepresence robot; A communication system according to any one of claims 1 to 5.

8. The other image display means has, as the editing process by the user, a two-stage editing process of editing three-dimensional illustration data and editing two-dimensional illustration data. The communication system of claim 1 .

9. The editing process for the three-dimensional illustration data is one of three-dimensional rotation, three-dimensional movement, three-dimensional enlargement, and three-dimensional reduction, The editing process for the two-dimensional illustration data is any one of two-dimensional rotation, two-dimensional movement, two-dimensional enlargement, two-dimensional reduction, erasure, addition of hand-drawn line drawings, and coloring. The communication system according to claim 8.

10. a language information input means for receiving input of language information; recognition means for recognizing input linguistic information as character information; an image display means for displaying an image corresponding to a keyword included in the character information on a display means for displaying an image based on a recognition result by the recognition means; a selection means for selecting the displayed image; another image display means for displaying an image corresponding to the selected image on the display means in a state in which an editing process operation by a user can be accepted; A display device comprising:

11. A display control method for a display device, comprising: a language information input step of accepting input of language information; a recognition step of recognizing input linguistic information as character information; an image display control step of displaying an image corresponding to a keyword included in the character information on a display means for displaying images based on a recognition result from the recognition step; a selection step of selecting the displayed image; another image display step of displaying an image corresponding to the selected image on the display means in a state in which an editing process operation by a user can be accepted; A display control method including:

12. Computer, a language information input means for receiving input of language information; recognition means for recognizing input linguistic information as character information; an image display means for displaying an image corresponding to a keyword included in the character information on a display means for displaying an image based on a recognition result by the recognition means; a selection means for selecting the displayed image; another image display means for displaying an image corresponding to the selected image on the display means in a state in which an editing process operation by a user can be accepted; A display control program that functions as a

Images