IMAGE EDITING DEVICE, IMAGE EDITING METHOD, AND IMAGE EDITING PROGRAM

Abstract

To provide a technique to present visual information that can be edited by a user.SOLUTION: An image editing device (server, client terminal) has: display means that displays image data; first editing means that executes first editing processing on first image data having a first data format of the image data displayed on the display means; second editing means that executes second editing processing on second image data having a second data format different from the first data format of the image data displayed on the display means; and conversion means that converts the first data format into the second data format. On the display means, when the first editing processing is executed by the first editing means, display of the first image data is changed to display of the image data after the first editing processing, and when the second editing processing is executed on the second image data obtained through conversion performed by the conversion means, display of the second image data is changed to display after the second editing processing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image editing device, an image editing method, and an image editing program. [Background technology]

[0002] In communication situations, verbal communication alone may not be sufficient to transmit information efficiently and accurately. For example, in meetings to think up new plans or ideas, it is effective to use visual information to communicate. In communication situations, accurate means of visual information are desired.

[0003] In Patent Document 1, an illustration displayed in the illustration search result display area is selected on the minutes editing screen of a graphic recording system, and the selected illustration is pasted into the minutes to create minutes that include illustrations. It is described that by looking at the minutes that include illustrations, it is possible to efficiently review the meeting. Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 only allows the selection of an illustration set in the graphic recording system, and depending on the illustration set, it may not be possible to adequately convey the user's intention.

[0005] The present invention has been made in view of the above, and an object of the present invention is to provide an image editing device, an image editing method, and an image editing program that are capable of presenting visual information that can be edited by a user. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, an image editing device according to one aspect of the present invention comprises a display means for displaying image data, a first editing means for performing a first editing process on first image data having a first data format among the image data displayed on the display means, a second editing means for performing a second editing process on second image data having a second data format different from the first data format among the image data displayed on the display means, and a conversion means for converting the first data format into the second data format, wherein when the first editing process is performed by the first editing means, the display of the first image data is changed to image data after the first editing process, and when the second editing process is performed on the second image data converted by the conversion means, the display of the second image data is changed to a display after the second editing process. Effect of the Invention

[0007] Advantageous Effects of Invention According to the present invention, it is possible to provide visual information that can be edited by a user. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a visual communication system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of a computer applied to the visual communication system according to the embodiment. [Diagram 3] FIG. 3 is a diagram showing a configuration of a language information input unit in the embodiment. [Figure 4] FIG. 4 is a diagram showing a configuration of an illustration display unit in the embodiment. [Diagram 5] FIG. 5 is a data flow diagram illustrating the conversion of image data in an embodiment. [Figure 6] FIG. 6 is a diagram showing a data structure of illustration information stored in the illustration storage unit in the embodiment. [Figure 7]FIG. 7 is a diagram showing a data structure of thumbnail information stored in a thumbnail storage unit in the embodiment. [Figure 8] FIG. 8 is a diagram showing the configuration of a drawing operation unit in the embodiment. [Figure 9] FIG. 9 is a flowchart illustrating the operation of the visual communication system according to the embodiment. [Figure 10] FIG. 10 is a flowchart showing the flow of the registration process in the embodiment. [Figure 11] FIG. 11 is a flowchart showing the flow of the communication support process in the embodiment. [Figure 12] FIG. 12 is a flowchart showing the flow of the communication support process in the embodiment. [Figure 13] FIG. 13 is a diagram showing a configuration of a display screen of the visual communication system according to the embodiment. [Figure 14] FIG. 14 is a diagram showing a schematic operation of a display screen of the visual communication system in the embodiment. [Figure 15] FIG. 15 is a diagram showing the flow of a two-stage editing process in the embodiment. [Figure 16] FIG. 16 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. [Figure 17] FIG. 17 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. [Figure 18] FIG. 18 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. [Figure 19] FIG. 19 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. [Figure 20] FIG. 20 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. [Figure 21]FIG. 21 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. [Figure 22] FIG. 22 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. [Figure 23] FIG. 23 is a diagram showing a detailed operation of a display screen by the visual communication system in the embodiment. [Figure 24] FIG. 24 is a diagram illustrating a configuration of an online conference system according to a modified example of the embodiment. [Diagram 25] FIG. 25 is a diagram showing a camera image captured by a camera of a client terminal in a modified example of the embodiment. [Figure 26] FIG. 26 is a diagram showing an image superimposed on the camera image of FIG. 25 by a drawing display unit in a modification of the embodiment. [Figure 27] FIG. 27 is a diagram showing an image displayed on another client terminal in the modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (Embodiment) The visual communication system according to the embodiment has a function of supporting communication. In communication situations, communication by language information alone may not be sufficient to efficiently and accurately transmit information. For example, in a meeting held when thinking of new plans or ideas, it is effective to use visual information for communication. One possible way to visualize information is to sketch a picture of an idea by hand, but it may not be easy to visualize the scene in one's head from scratch. For this reason, a first technology is known that converts information into visual information based on language information that is easy for anyone to use.

[0010] However, the first technology, which converts language information into visual information, only searches for illustrations prepared in advance. To be more specific, there are many cases where the illustrations that match the image in one's head and the image combination do not exist, making it difficult to properly communicate information. Increasing the number of prepared illustrations increases the probability that an illustration matching the image in one's head exists, but as the number of illustrations increases, the time required to select increases. Also, even if one tries to edit the illustrations themselves in order to accurately reproduce the image in one's head, if there is only a simple editing function, the edited illustration is unlikely to be close to the image in one's head. Therefore, the first technology may not be able to accurately communicate information in communication situations such as meetings.

[0011] Therefore, in this embodiment, in a visual communication system, an illustration corresponding to linguistic information received from a user is searched for and displayed, and a two-stage editing process is enabled by changing the display form of the displayed illustration, thereby supporting accurate information transmission in communication situations.

[0012] Specifically, the visual communication system converts language information into visual information that can be edited as desired, and enables accurate communication of the scene in one's head in a short time in a communication situation such as a conference. The visual communication system analyzes language information such as speech and character input in a communication situation in real time, and displays one or more thumbnail images in a specific area as candidates for illustrations related to the language information at the same time as the analysis. The visual communication system has first correspondence information in which language information and thumbnail images are associated, and when language information is received, one or more thumbnail images corresponding to the language information in the first correspondence information are specified and displayed. In addition, the visual communication system has second correspondence information in which thumbnail images are associated with illustrations. When the illustration is a three-dimensional image, the visual communication system can display the thumbnail image at a higher speed than when displaying an illustration by making the thumbnail image a two-dimensional image. Each time language information from a user is received, the thumbnail image on the display screen can be switched to a new one and updated and displayed at a high speed. This allows the user to select one or more thumbnail images that are close to the scene in his or her head when the speech or character input is received from among them without stopping the conversation. According to the second correspondence information, three-dimensional illustration data associated with the selected thumbnail image is searched for, and the identified three-dimensional illustration data is displayed. This allows three-dimensional editing (three-dimensional movement, enlargement, reduction, and rotation) of the three-dimensional illustration data. After that, the visual communication system converts (fixes) the three-dimensional illustration data that can be three-dimensionally edited into two-dimensional illustration data that can be two-dimensionally edited. In response to this, two-dimensional editing (freely erasing parts, adding on top, or painting in color) of the two-dimensional illustration data is possible. This two-stage editing process triggered by automatic recognition of language information allows the scene in one's head to be accurately expressed in a short time in a real-time communication situation, enabling rapid and accurate communication.

[0013] More specifically, the visual communication system 4 may be configured as shown in Fig. 1. Fig. 1 is a diagram showing the configuration of the visual communication system 4.

[0014] The visual communication system 4 includes a client terminal 1, a server 2, and a connection unit 3. The connection unit 3 connects the client terminal 1 and the server 2 so that they can communicate with each other.

[0015] Client terminal 1 has a language information input unit 100, an illustration display unit 200, and a drawing operation unit 300. Server 2 has a drawing display unit 400 and a storage unit 500. Storage unit 500 stores a program 500a.

[0016] When visual communication system 4 receives a startup request from a user at client terminal 1, the startup request is transmitted from client terminal 1 to server 2, and in response to the startup request, server 2 reads program 500a from storage unit 500. In accordance with program 500a, visual communication system 4 functionally configures language information input unit 100, illustration display unit 200, and drawing operation unit 300 in client terminal 1, and functionally configures drawing display unit 400 in server 2, as shown in FIG.

[0017] In addition, in the visual communication system 4, the illustration display unit 200 and the drawing operation unit 300 in the client terminal 1 may be configured in the server 2 or another server. Alternatively, the visual communication system 4 may be configured so that all the functional configurations including the language information input unit 100, the illustration display unit 200, the drawing operation unit 300, and the drawing display unit 400 are completed in the client terminal 1. Alternatively, the visual communication system 4 may be configured so that a part of the elements included in the illustration display unit 200 (for example, a user interface and a part similar thereto) are configured in the client terminal 1, and the remaining part is configured in the server 2 or another server. Similarly, the visual communication system 4 may be configured so that a part of the elements included in the drawing operation unit 300 (for example, a user interface and a part similar thereto) are configured in the client terminal 1, and the remaining part is configured in the server 2 or another server.

[0018] The connection unit 3 may be a wired communication line and / or a wireless communication line, may be a so-called communication network, may be a communication cable, etc. The connection unit 3 may be constructed using one or more of the Internet, a mobile communication network, a LAN (Local Area Network), etc. The connection unit 3 may include not only wired communication but also a network using wireless communication such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution). When the program 500a is stored in the client terminal 1 and each functional configuration is configured to be completed within the client terminal 1, the connection unit 3 may be omitted.

[0019] In the client terminal 1, the language information input unit 100 as a language information input means accepts input of language information by the user. The illustration display unit 200 displays images such as characters and illustrations on the display. The drawing operation unit 300 accepts drawing operations by the user. Here, drawing includes not only handwriting but also a selection operation on the display for the purpose of drawing on the display. The drawing operation unit 300 of the client terminal 1 transmits a drawing operation request to the drawing display unit 400 of the server 2. In response to the drawing operation request, the drawing display unit 400 of the server 2 updates the display information so as to change the display form of the displayed image, and transmits the updated display information to the client terminal 1. The client terminal 1 receives the updated display information and displays an image according to the display information on the display. As a result, the result of the drawing operation by the user is displayed on the display of the client terminal 1.

[0020] Each of the client terminal 1 and the server 2 may be configured in terms of hardware using a computer 5 as shown in Fig. 2. Fig. 2 is a diagram showing the hardware configuration of the computer 5 applied to the visual communication system 4.

[0021] As shown in FIG. 2, the computer 5 is equipped with a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, a media I / F 516, a motion acquisition device 517, a microphone 518, a speaker 519, and a camera 520.

[0022] Among these, the CPU 501 controls the operation of the entire computer 5. The ROM 502 stores programs such as an IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as the program 500a. The HDD controller 505 controls the reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 as a display means displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using the connection unit 3. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 2.

[0023] Moreover, the keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513 as an example of a removable recording medium. Note that this is not limited to DVD-RW, and may be DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515 such as a flash memory.

[0024] The motion acquisition device 517 is a circuit that detects the user's motion and converts it into an electrical signal, and is a type of input means. The user's motion may be detected by any of the following means: an optical type that detects the reflection of light; a magnetic sensor, a mechanical sensor, or a magnetic sensor attached to the user; a method that analyzes an image of the user; or a combination of these.

[0025] The microphone 518 is a built-in circuit that converts sound into an electrical signal. The speaker 519 is a built-in circuit that converts the electrical signal into physical vibrations to produce sound such as music or voice.

[0026] The speaker 519 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice.

[0027] The camera 520 is a type of built-in imaging means that captures an image of a subject and obtains image data.

[0028] In the computer 5 applied to the server 2, the HD 504 corresponds to the storage unit 500 and stores the program 500a. In the computer 5 applied to the client terminal 1, the program 500a is downloaded from the server 2 via the connection unit 3 and stored in the HD 504, and the program 500a is read from the HD 504 and executed by the CPU 501, so that the functional configuration as shown in Fig. 1 can be expanded in the RAM 503 either collectively at the time of compilation or sequentially as the processing progresses. In the computer 5 applied to the server 2, the program 500a is executed by the CPU 501, so that the functional configuration as shown in Fig. 1 can be expanded in the RAM 503 either collectively at the time of compilation or sequentially as the processing progresses.

[0029] Furthermore, the computer 5 applied to the server 2 may omit configuration related to the user interface, and at least one of the display 506, the keyboard 511, the pointing device 512, and the action acquisition device 517 may be omitted.

[0030] In addition, the client terminal 1 and the server 2 may be, other than a computer 5, an IWB (Interactive White Board: an electronic whiteboard function capable of intercommunication), an output device such as digital signage, a HUD (Head Up Display) device, industrial machinery, medical equipment, network home appliances, automobiles (Connected Cars), mobile phones, smartphones, tablet terminals, game consoles, PDAs (Personal Digital Assistants), etc.

[0031] The language information input unit 100 shown in Fig. 1 can be functionally configured as shown in Fig. 3. Fig. 3 is a diagram showing the configuration of the language information input unit 100.

[0032] The language information input unit 100 includes a voice input unit 101, a character input unit 102, a recognition unit 103, and a transmission unit 104. Each of these units is a function or means realized by operating according to an instruction from a CPU 501 in accordance with a program loaded onto a RAM 503 from a HD 504.

[0033] The voice input unit 101 receives, as language information, a voice signal obtained by converting a user's voice using the microphone 318, or a voice signal received via the network I / F 509. The voice input unit 101 supplies the voice signal to the recognition unit 103.

[0034] Character input unit 102 accepts, as language information, a character signal input by a user via keyboard 541, pointing device 512, or motion acquisition device 517, or a character signal received via a network I / F. Character input unit 102 supplies the character signal to recognition unit 103. It is assumed that characters are input to character input unit 102 by typing on keyboard 541 or by handwriting input using pointing device 512 or motion acquisition device 517.

[0035] Upon receiving linguistic information from the speech input unit 101 or the character input unit 102, the recognition unit 103 recognizes character information for the linguistic information.

[0036] When the recognition unit 103 receives a voice signal from the voice input unit 101, it performs voice recognition processing on the voice signal and converts it into character information. The recognition unit 103 has a template voice signal for each character. The recognition unit 103 performs matching processing on the voice signal using the template voice signal, and can recognize characters for each voice based on the matching score. As a result, the recognition unit 103 generates character information as a recognition result for the voice signal.

[0037] When the recognition unit 103 receives a character signal from the character input unit 102, it generates character information as a recognition result for the character signal. When the recognition unit 103 receives character information typed on the keyboard 541, it treats the character information as a recognition result. When the recognition unit 103 receives a handwritten character image by the pointing device 512 or the motion acquisition device 517, it performs text recognition processing on the handwritten character image and converts it into character information. The recognition unit 103 has a template character image for each character. The recognition unit 103 performs matching processing on the handwritten character image using the template character image, and can recognize characters for each handwritten character image based on the matching score. In this way, the recognition unit 103 generates character information as a recognition result for the handwritten character image.

[0038] The recognition unit 103 supplies the recognition result to the transmission unit 104. The transmission unit 104 transmits the recognition result to the illustration display unit 200.

[0039] The illustration display section 200 shown in Fig. 1 can be functionally configured as shown in Fig. 4. Fig. 4 is a diagram showing the configuration of the illustration display section 200.

[0040] The illustration display unit 200 comprises a receiving unit 201, a thumbnail search unit 202, a thumbnail display unit 203, a thumbnail selection unit 204, an illustration search unit 205, an illustration display unit 206, a context analysis unit 207, a priority determination unit 208, a thumbnail creation unit 211, an illustration creation unit 213, a three-dimensional data input unit 214, a keyword creation unit 215, and a registered character input unit 216.

[0041] These units are functions or means that are realized by operating according to instructions from the CPU 501 in accordance with the program 500 a loaded onto the RAM 503 from the HD 504 .

[0042] Moreover, the illustration display unit 200 includes a selection tendency storage unit 209, a thumbnail storage unit 210, and an illustration storage unit 212. Each of these units is constructed by the ROM 502, the RAM 503, or the HD 504.

[0043] The data storage in the thumbnail storage section 210 and the illustration storage section 212 will be described with reference to Figures 4 and 5. Figure 5 is a data flow diagram showing the conversion of image data.

[0044] The three-dimensional data input unit 214 receives three-dimensional data. The three-dimensional data is, for example, polygon data as shown in FIG. 5(a) and includes a plurality of spatial coordinates. The three-dimensional data corresponds to a three-dimensional image data format (for example, OBJ format). The three-dimensional data input unit 214 supplies the three-dimensional data to the illustration unit 213.

[0045] The illustration conversion unit 213 converts the three-dimensional data into three-dimensional illustration data. The three-dimensional illustration data is, for example, three-dimensional line drawing data as shown in FIG. 5(b) and includes a plurality of spatial coordinates. The three-dimensional illustration data corresponds to a three-dimensional image data format (for example, OBJ format). The illustration conversion unit 213 extracts line drawing information from the three-dimensional data by identifying the three-dimensional positions of the edges of the three-dimensional shape represented by the three-dimensional data and connecting them with line drawing, thereby generating three-dimensional illustration data. The illustration conversion unit 213 additionally stores the three-dimensional illustration data in the illustration storage unit 212.

[0046] As a result, illustration information 212a as shown in Fig. 6 is stored in illustration storage unit 212. Fig. 6 is a diagram showing the data structure of illustration information 212a stored in illustration storage unit 212. In illustration information 212a, three-dimensional illustration data and its identification information are associated with one or more pieces of three-dimensional illustration data. For example, illustration information 212a has an identification information field 212a1 and an access information field 212a2. In identification information field 212a1, information for identifying the three-dimensional illustration data is recorded, for example, an ID number of the three-dimensional illustration data is recorded. In access information field 212a2, information for accessing the three-dimensional illustration data is recorded, for example, a file name of the three-dimensional illustration data is recorded.

[0047] The thumbnail generating unit 211 acquires the three-dimensional illustration data from the illustration storage unit 212 at the timing when the three-dimensional illustration data is added to the illustration storage unit 212 or at predetermined intervals, and converts the three-dimensional illustration data into two-dimensional thumbnail data. The two-dimensional thumbnail data is, for example, two-dimensional line drawing data as shown in FIG. 5(c), and is line drawing data equivalent to the three-dimensional illustration data being reduced and two-dimensionalized. The two-dimensional thumbnail data corresponds to a two-dimensional image data format (for example, BMP format). The two-dimensional thumbnail data is associated with identification information of the three-dimensional illustration data. The thumbnail generating unit 211 obtains a plurality of plane coordinates in which a plurality of spatial coordinates included in the three-dimensional illustration data are projected onto a predetermined plane, and reduces the two-dimensional line drawing data corresponding to the obtained plurality of plane coordinates, thereby generating two-dimensional thumbnail data that has been reduced and two-dimensionalized from the three-dimensional illustration data. The thumbnail generating unit 211 additionally stores the two-dimensional thumbnail data in the thumbnail storage unit 210 in a form associated with the identification information of the three-dimensional illustration data.

[0048] As a result, thumbnail information 210a as shown in FIG. 7 is stored in thumbnail storage unit 210. FIG. 7 is a diagram showing a data structure of thumbnail information 210a stored in thumbnail storage unit 210. Thumbnail information 210a is information in which a keyword, three-dimensional illustration data, and two-dimensional thumbnail data are associated with one or more pieces of two-dimensional thumbnail data. Thumbnail information 210a includes information in which a keyword and two-dimensional thumbnail data are associated with each other as first association information. As shown in FIG. 7, a plurality of keywords may be set for one piece of two-dimensional thumbnail data. Furthermore, thumbnail information 210a includes information in which two-dimensional thumbnail data and three-dimensional illustration data are associated with each other as second association information. For example, thumbnail information 210a has a keyword column 210a1, an identification information column 210a2, and an access information column 210a3. A keyword for calling up two-dimensional thumbnail data is recorded in keyword column 210a1, but the column is blank when no keyword is registered. The identification information field 210a2 stores information for identifying the 3D illustration data associated with the 2D thumbnail data, such as an ID number for the 3D illustration data. The access information field 210a3 stores information for accessing the 2D thumbnail data, such as a file name for the 2D thumbnail data.

[0049] Character information to be linked to the 3D illustration data is input to registered character input unit 216 at the timing when 3D illustration data is added to illustration accumulation unit 212 or at predetermined intervals. Registered character input unit 216 accepts the character information in a form associated with identification information of the 3D illustration data. At this time, registered character input unit 216 may acquire identification information of multiple 3D illustration data that are candidates to be linked to the character information from illustration accumulation unit 212, as indicated by the dotted arrow, and selectively input identification information from the identification information of the multiple 3D illustration data. Registered character input unit 216 supplies the character information to keyword generation unit 215 in a form associated with the identification information of the 3D illustration data.

[0050] When keyword conversion unit 215 receives the text information associated with the identification information of the three-dimensional illustration data, it accesses thumbnail storage unit 210 and additionally writes the text information into keyword field 210a1 corresponding to the identification information of the three-dimensional illustration data.

[0051] As a result, in thumbnail information 210a stored in thumbnail storage section 210, one or more keywords are additionally recorded in keyword column 210a1, as shown in FIG.

[0052] The data size of the 2D thumbnail data is significantly smaller than that of the 3D illustration data. By using thumbnails with a small data size when searching for illustrations based on language information, it is possible to minimize the time lag when retrieving images.

[0053] The data storage in the selection tendency storage unit 209 will be described with reference to FIG. 4. When the receiving unit 201 receives the language information recognition result, it supplies the language information recognition result to the context analysis unit 207. The context analysis unit 207 analyzes the context based on the language information recognition result. Furthermore, the selection information in the thumbnail selection unit 204 is stored in the selection tendency storage unit 209 as the user's selection tendency. Based on the analysis result of the context analysis unit 207 and the storage result of the selection tendency storage unit 209, the priority determination unit 208 determines the priority when displaying the 2D thumbnail data, and supplies the determination result to the thumbnail display unit 203. The thumbnail display unit 203 displays the 2D thumbnail data on the display in descending order of priority according to the determination result of the priority determination unit 208.

[0054] An example of the priority may be a priority determined in response to a foreign language as language information. The priority determination unit 208 may determine a priority such that the thumbnail to be displayed is changed according to the type of language. For example, when the word "school" is detected in an African language, the priority is determined so that an illustration showing a school commonly seen in Africa is preferentially displayed instead of an illustration showing a typical school in Japan.

[0055] A method of displaying an illustration will be described with reference to FIG. 4. When the receiving unit 201 receives the language information recognition result, it supplies the language information recognition result to the thumbnail search unit 202. The thumbnail search unit 202 searches for two-dimensional thumbnail data based on the language information recognition result. The thumbnail search unit 202 accesses the thumbnail storage unit 210, searches for thumbnail information 210a using character information corresponding to the language information recognition result as a keyword, retrieves one or more two-dimensional thumbnail data corresponding to the character information (e.g., keyword) and identification information of three-dimensional illustration data associated therewith as search results, and supplies them to the thumbnail display unit 203. The thumbnail display unit 203, which serves as a first display control means, supplies the retrieved one or more two-dimensional thumbnail data to the drawing display unit 400 (see FIG. 1) via the connection unit 3. In response to this, the drawing display unit 400 displays the retrieved one or more two-dimensional thumbnail data at a specific location on the display of the client terminal 1. Thumbnail selection unit 204 as a selection means selects two-dimensional thumbnail data from one or more two-dimensional thumbnail data displayed on the display in response to a selection operation by a user, for example, an operation of selecting two-dimensional thumbnail data displayed on the display with a pointing device, and supplies identification information of three-dimensional illustration data associated with the selected two-dimensional thumbnail data to illustration search unit 205. Illustration search unit 205 searches for three-dimensional illustration data based on the identification information of the three-dimensional illustration data. Illustration search unit 205 accesses illustration storage unit 212, retrieves three-dimensional illustration data corresponding to the identification information as a search result, and supplies the data to illustration display unit 206. Illustration display unit 206 as a second display control means supplies the searched three-dimensional illustration data to drawing display unit 400 (see FIG. 1) via connection unit 3. In response to this, drawing display unit 400 causes the searched three-dimensional illustration data to be displayed on the display of client terminal 1.

[0056] The drawing operation unit 300 shown in Fig. 1 can be functionally configured as shown in Fig. 8. Fig. 8 is a diagram showing the configuration of the drawing operation unit 300.

[0057] The drawing operation unit 300 includes an illustration editing unit 301 , a fixing unit 302 , a drawing unit 303 , a hand-drawn input unit 304 , and an output unit 305 .

[0058] The illustration editing unit 301 as a first editing means (three-dimensional data editing input means) receives three-dimensional illustration data from the illustration display unit 200 and performs three-dimensional editing processing on the three-dimensional illustration data. In the three-dimensional editing processing, the illustration editing unit 301 accepts a three-dimensional rotation operation, a three-dimensional movement operation, a three-dimensional enlargement operation, a three-dimensional reduction operation, and the like by the user, and supplies these operation requests to the drawing display unit 400 (see FIG. 1) via the drawing unit 303, the output unit 305, and the connection unit 3. The three-dimensional rotation operation is an operation in which a plurality of spatial coordinates included in the three-dimensional illustration data are changed so as to be rotated three-dimensionally around a predetermined axis while maintaining their relative positional relationship. The three-dimensional enlargement operation is an operation in which a plurality of spatial coordinates included in the three-dimensional illustration data are changed so as to move radially away from a predetermined point at an equal distance rate. The three-dimensional reduction operation is an operation in which a plurality of spatial coordinates included in the three-dimensional illustration data are changed so as to move radially closer to a predetermined point at an equal distance rate. In response to these operational requests, drawing display unit 400 changes the display form of the 3D illustration data in drawing unit 303 of client terminal 1. This allows the position, size, and orientation of the 3D illustration data on the display of client terminal 1 to be changed three-dimensionally.

[0059] The fixing unit 302 as a conversion means converts (fixes) three-dimensional illustration data that can be edited in a three-dimensional manner into two-dimensional illustration data that can be edited in a two-dimensional manner in response to a predetermined operation (for example, an operation that activates a two-dimensional editing process, more specifically, an operation that transitions to a state where a two-dimensional editing process is possible). The two-dimensional illustration data is two-dimensional line drawing data as shown in FIG. 5(d), and is line drawing data that corresponds to the three-dimensional illustration data that has been subjected to a three-dimensional editing process and turned into two-dimensional data. The two-dimensional illustration data corresponds to a two-dimensional image data format (for example, BMP format). The fixing unit 302 converts the three-dimensional illustration data into two-dimensional illustration data by obtaining a plurality of plane coordinates in which a plurality of spatial coordinates included in the three-dimensional illustration data are projected onto a plane corresponding to the display screen, and generates two-dimensional line drawing data corresponding to the obtained plurality of plane coordinates, thereby converting the three-dimensional illustration data into two-dimensional illustration data and fixing it to the drawing unit 303.

[0060] The two-dimensional data edit input unit 304 as a second editing means (two-dimensional data edit input means) performs two-dimensional editing processing on the two-dimensional illustration data. In the two-dimensional editing processing, the two-dimensional data edit input unit 304 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations of adding hand-drawn lines, coloring operations, and the like, performed by the user, and supplies these operation requests to the drawing display unit 400 (see FIG. 1) via the drawing unit 303, the output unit 305, and the connection unit 3. The two-dimensional rotation operation is an operation in which a plurality of planar coordinates included in the two-dimensional illustration data are changed so as to rotate two-dimensionally around a specified point while maintaining their relative positional relationship. The two-dimensional enlargement operation is an operation in which a plurality of planar coordinates included in the two-dimensional illustration data are changed so as to move radially away from a specified point at equal distance ratios. The two-dimensional reduction operation is an operation in which a plurality of plane coordinates included in the two-dimensional illustration data are changed so as to approach a predetermined point radially at an equal distance ratio. The partial erasure operation is an operation in which a portion of a plurality of plane coordinates included in the two-dimensional illustration data is deleted. The operation of adding handwritten lines is an operation in which a plurality of plane coordinates corresponding to handwritten lines are added to a plurality of plane coordinates included in the two-dimensional illustration data. The operation of coloring is an operation in which a plurality of plane coordinates associated with a predetermined color attribute are added to a plurality of plane coordinates included in the two-dimensional illustration data. In response to these operation requests, the drawing display unit 400 changes the display form of the two-dimensional illustration data in the drawing unit 303 of the client terminal 1. As a result, the position, size, and orientation of the two-dimensional illustration data on the display of the client terminal 1 are changed two-dimensionally, a portion of the two-dimensional illustration data is erased, handwritten lines are added to the two-dimensional illustration data, or the two-dimensional illustration data is colored.

[0061] Input to the drawing unit 303 can also be made by a mouse, a finger, a stylus pen, gestures, etc. Information from the drawing unit 303 is output by the output unit 305 to the drawing display unit 400. In response to this, the drawing display unit 400 adds line drawings, etc., handwritten and input in the drawing unit 303 of the client terminal 1. As a result, the handwritten and input line drawings, etc. are displayed on the display of the client terminal 1.

[0062] Each of these units is a function or means realized by operating according to an instruction from the CPU 501 in accordance with a program loaded onto the RAM 503 from the HD 504 .

[0063] Next, the operation of the visual communication system 4 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the operation of the visual communication system.

[0064] The visual communication system 4 performs a registration process (S1) in which predetermined information is registered in preparation for assisting communication. Thereafter, the visual communication system 4 waits until there is a start request from the user (No in S2). When there is a start request from the user (Yes in S2), the visual communication system 4 starts the program 500a and starts the communication assistance process (S3). The visual communication system 4 continues to perform the communication assistance process (S3) until there is an end request from the user (No in S4). When there is an end request from the user (Yes in S4), the visual communication system 4 ends the process.

[0065] Next, details of the registration process (S1) will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the registration process.

[0066] When program 500a is started, visual communication system 4 waits until there is a request to register three-dimensional data (No in S11). When there is a request to register three-dimensional data (Yes in S11), visual communication system 4 inputs the three-dimensional data (S12). The three-dimensional data is, for example, polygon data (see FIG. 5(a)). Visual communication system 4 converts the three-dimensional data into three-dimensional illustration data (S13). That is, visual communication system 4 converts the three-dimensional data into three-dimensional illustration data. The three-dimensional illustration data is, for example, three-dimensional line drawing data (see FIG. 5(b)). Visual communication system 4 additionally stores the three-dimensional illustration data in illustration storage unit 212. As a result, illustration information 212a (see FIG. 6) is registered in illustration storage unit 212.

[0067] The visual communication system 4 thumbnails the three-dimensional illustration data (S14). That is, the visual communication system 4 converts the three-dimensional illustration data into two-dimensional thumbnail data. The two-dimensional thumbnail data is, for example, two-dimensional line drawing data (see FIG. 5(c)). The visual communication system 4 additionally stores the two-dimensional thumbnail data in thumbnail storage unit 210 in a form associated with identification information of the three-dimensional illustration data. As a result, the two-dimensional thumbnail data and the identification information of the three-dimensional illustration data in thumbnail information 210a (see FIG. 7) are registered in thumbnail storage unit 210.

[0068] Then, the visual communication system 4 waits until a keyword registration request is received (No in S15). When the visual communication system 4 receives a keyword registration request (Yes in S15), character information is input in a form associated with the identification information of the three-dimensional illustration data (S16). The visual communication system 4 converts the character information into a keyword (S17). That is, the visual communication system 4 accesses the thumbnail storage unit 210 and additionally writes the character information in the keyword field 210a1 corresponding to the identification information of the three-dimensional illustration data. As a result, the character information (keyword) in the thumbnail information 210a (see FIG. 7) is registered in the thumbnail storage unit 210. Note that it is not necessary to perform both the registration process and the communication support process as shown in FIG. 9, and it is also possible to execute only the communication support process based on the information registered in the thumbnail storage unit 210 and the illustration storage unit 212 at that time.

[0069] Next, details of the communication support process (S3) will be described with reference to Fig. 11 and Fig. 12. Fig. 11 and Fig. 12 are flowcharts showing the flow of the communication support process. The process of Fig. 11 and the process of Fig. 12 can be performed in parallel with each other.

[0070] In the process of FIG. 11, when the program 500a is started and an initial screen is displayed on the display, the visual communication system 4 waits until there is a request to turn on the language information input function via the initial screen (No in S21). When there is a request to turn on the language information input function (Yes in S21), the visual communication system 4 waits until language information is input (No in S22). When language information is input (Yes in S22), the visual communication system 4 judges whether the language information is voice information or not (S23). If the language information is voice information (Yes in S23), the visual communication system 4 performs voice recognition processing on the voice information to convert it into character information (S24), and the character information is the recognition result. If the language information is not voice information (No in S23), the visual communication system 4 judges whether the language information is a handwritten character image or not (S25). If the language information is a handwritten character image (Yes in S25), the visual communication system 4 performs text recognition processing on the handwritten character image to convert it into character information (S26) and sets the character information as the recognition result. If the language information is not a handwritten character image, that is, if the language information is character information typed on the keyboard 541 (No in S25), the visual communication system 4 sets the character information as the recognition result and proceeds to S27.

[0071] The visual communication system 4 accesses the thumbnail storage unit 210 and searches for 2D thumbnail data using the character information that is the recognition result as a keyword (S27). The visual communication system 4 displays one or more pieces of searched 2D thumbnail data in a specific location on the display (S28).

[0072] The visual communication system 4 can quickly repeat the processes of S22 to S28 until there is a request to turn off the language information input function (No in S29). That is, the visual communication system 4 can quickly update and display new thumbnail images on the display screen every time language information is received from the user. This allows the user to select one or more thumbnail images that are closest to the scene in their mind when they spoke or entered text, without stopping the conversation.

[0073] When a request to turn off the language information input function is received (Yes in S29), the visual communication system 4 ends the process of FIG.

[0074] 12, visual communication system 4 waits until 2D thumbnail data is selected from the one or more pieces of 2D thumbnail data displayed in S28 (No in S31). When 2D thumbnail data is selected from the one or more pieces of 2D thumbnail data displayed in S28 (Yes in S31), visual communication system 4 accesses illustration storage unit 212 and searches for 3D illustration data associated with the selected 2D thumbnail data (S32). Visual communication system 4 displays the searched 3D illustration data on the display (S33).

[0075] The visual communication system 4 waits until an editing operation is performed on the three-dimensional illustration data (No in S34), and when an editing operation is performed on the three-dimensional illustration data (Yes in S34), it performs a first editing process (S35). The first editing process is a three-dimensional editing process. In the first editing process, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, and the like. In response to these operation requests, the visual communication system 4 changes the display form of the three-dimensional illustration data displayed on the display (S36). The visual communication system 4 repeats the processes of S34 to S36 until the three-dimensional illustration data is fixed to two-dimensional illustration data (No in S37).

[0076] When the three-dimensional illustration data is fixed to two-dimensional illustration data (Yes in S37), the visual communication system 4 waits until an editing operation is performed on the two-dimensional illustration data (No in S38). When an editing operation is performed on the two-dimensional illustration data (Yes in S38), the visual communication system 4 performs a second editing process (S39). The second editing process is a two-dimensional editing process. In the second editing process, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn lines, coloring operations, and the like. In response to these operation requests, the visual communication system 4 changes the display form of the two-dimensional illustration data displayed on the display (S40). The visual communication system 4 repeats the processes of S38 to S40 until a request to complete editing is received (No in S41). That is, by the first stage of editing processing in the loop of S34 to S37 and the second stage of editing processing in the loop of S38 to S41, an illustration close to the scene in the mind when the speech or character input is performed can be accurately expressed in a short time.

[0077] When a request for completing editing is received (Yes in S41), the visual communication system 4 ends the processing in FIG.

[0078] Next, the configuration of a screen displayed on the display of the client terminal 1 by the visual communication system 4 (a display screen by the visual communication system 4) will be described with reference to Fig. 13. Fig. 13 is a diagram showing the configuration of a display screen by the visual communication system 4.

[0079] As shown in FIG. 13, the display screen of the visual communication system 4 includes a drawing result display area S100, an operation palette S200, and a thumbnail display area S300.

[0080] The drawing result display area S100 is an area for outputting the results of the drawing display unit 400 (see FIG. 1). The operation palette S200 has functions arranged therein for calling up each input unit, such as the voice input unit 101, the character input unit 102 (see FIG. 3), and the hand-drawn input unit 304 (see FIG. 8). The thumbnail display area S300 displays the language information recognized by the recognition unit 103 (see FIG. 3) and the two-dimensional thumbnail data output by the thumbnail display unit 203 (see FIG. 4).

[0081] The layout and design of each area shown in Figure 5 are merely examples and shall not limit the scope of rights. For example, eight thumbnails surrounded by circular frames are displayed in S300, but there are no restrictions on the presence or absence of frames, their shapes, or the number of thumbnails displayed. In addition, recognized language information does not have to be displayed.

[0082] Next, the general operation of the screen (display screen by the visual communication system 4) displayed on the display of the client terminal 1 by the visual communication system 4 will be described with reference to Fig. 14. Fig. 14 is a diagram showing the general operation of the display screen by the visual communication system 4. Fig. 14 shows the operation of calling up 2D thumbnail data based on language information.

[0083] When the visual communication system 4 detects an operation of pressing the language information recognition button S201 shown in Fig. 14(a), the visual communication system 4 transitions to a language information recognition mode. In the language information recognition mode, when the visual communication system 4 acquires language information (first language information) by speech, handwriting, typing, or the like, the language information recognition result and one or more pieces of 2D thumbnail data linked to the recognition result are displayed in the thumbnail display area S300. In addition, when the visual communication system 4 detects an operation of pressing the 2D data edit button S202, the visual communication system 4 transitions to a drawing result edit mode in which the drawing result display area S100 in Fig. 13 can be edited.

[0084] For example, FIG. 14(a) shows the visual communication system 4 recognizing handwritten characters drawn on the drawing result display area S100, and displaying the character information of the recognition result and one or more pieces of two-dimensional thumbnail data associated with it in the thumbnail display area S300.

[0085] When the visual communication system 4 further acquires language information (second language information) by speaking, handwriting, typing, etc., at least a portion of the one or more two-dimensional thumbnail data displayed in the thumbnail display area S300 is changed, and the language information recognition result and the one or more two-dimensional thumbnail data linked to the recognition result are updated and displayed.

[0086] For example, when an utterance as shown in Fig. 14(b) is recognized in the state of Fig. 14(a), the visual communication system 4 changes the state of the thumbnail display area S300 in real time according to the character information of the recognition result. Fig. 14(b) shows a state in which the visual communication system 4 updates and displays in the thumbnail display area S301 based on the character information of the new recognition result and one or more pieces of two-dimensional thumbnail data associated with it.

[0087] Specifically, at least a portion of the thumbnail images already displayed is changed and displayed on the display (display means) in accordance with the thumbnail images corresponding to the newly input language information (second language information).

[0088] Changing at least a portion of the display of a thumbnail image already displayed means, for example, changing the position or size of a thumbnail image already displayed in order to add a thumbnail image corresponding to newly input language information to the already displayed thumbnail image and display it in place of the already displayed image, deleting the thumbnail image corresponding to newly input language information to display it in place of the already displayed image, changing the image surrounding the already displayed thumbnail image, etc.

[0089] The visual communication system 4 may change the display in the thumbnail display area S300 not only when speech is received but also whenever new handwritten or typed characters are recognized.

[0090] Furthermore, when the 2D thumbnail data cannot all be displayed in the thumbnail display area S300, the visual communication system 4 may display the data in the thumbnail display area S300 in order from oldest to newest retrieved 2D thumbnail data.

[0091] In FIG. 14, the switch for switching to the language information recognition mode is the language information recognition button S201, but the method is not limited thereto and may be, for example, speaking specific words or inputting a command instead of using a button.

[0092] 14, the visual communication system 4 can quickly update and display new thumbnail images on the display screen every time language information is received from the user. This allows the user to select one or more thumbnail images that are closest to the scene in his or her mind when speaking or inputting text, without stopping the conversation.

[0093] Next, the flow of the two-stage illustration editing process by the visual communication system 4 will be described with reference to Fig. 15. Fig. 15 is a diagram showing the flow of the two-stage editing process.

[0094] In FIG. 15(a), the visual communication system 4 displays, in response to the linguistic information "person", character information LI of "person" and two-dimensional thumbnail data SM1 to SM6 linked thereto in the thumbnail display area S300.

[0095] When the 2D thumbnail data SM5 is selected, the visual communication system 4 calls up the 3D illustration data linked to the 2D thumbnail data SM5 and displays it in the drawing result display area S100, as shown in FIG. 15(b).

[0096] In the three-dimensional editing process, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, etc., and in response to these operation requests, changes the display form of the three-dimensional illustration data three-dimensionally, as shown in Figure 15(c).

[0097] Upon receiving a predetermined trigger operation (e.g., pressing a button for a two-dimensional editing operation (e.g., the two-dimensional data editing button S202 shown in FIG. 14(a))), the visual communication system 4 fixes the three-dimensional illustration data to two-dimensional illustration data, as shown in FIG. 15(d).

[0098] In the two-dimensional editing process, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn line drawings, coloring operations, etc., and in response to these operation requests, changes the display form of the two-dimensional illustration data two-dimensionally, as shown in Figure 15(e).

[0099] As illustrated in FIG. 15, the first stage editing process (FIG. 15(b) and FIG. 15(c)) and the second stage editing process (FIG. 15(e)) make it possible to quickly and accurately express an illustration that closely resembles the scene that occurs in the mind when speech or text is input.

[0100] Next, detailed operations of the screens (display screens by the visual communication system 4) displayed on the display of the client terminal 1 by the visual communication system 4 will be described with reference to Figs. 16 to 23. Figs. 16 to 23 are diagrams each showing detailed operations of the display screens by the visual communication system 4. Fig. 14 shows operations in which two-dimensional thumbnail data is called based on language information, a thumbnail is selected, and illustration editing is performed.

[0101] When it is detected that the voice input button S2011 in the operation palette S200 shown in Fig. 16(a) has been pressed, the visual communication system 4 turns on the language information recognition function, and displays an icon S3021 indicating that voice input is in a standby state in the thumbnail display area S300, as shown in Fig. 16(b). When the key input button 2012 in the operation palette S200 is pressed, typing input via the keyboard becomes possible, and an icon indicating that typing input is in a standby state is displayed. These icons S3021 and the icon indicating that typing input is in a standby state are examples of language information recognition state display icons.

[0102] When it is detected that "people and vehicles" has been spoken, the visual communication system 4 displays the character information LI1 of "people and vehicles" and the two-dimensional thumbnail data SM11-SM18 linked thereto in the thumbnail display area S300, as shown in Fig. 17. Note that the thumbnail data linked to the character information may be thumbnail data linked to the entire character information of "people and vehicles", or may be a group of thumbnail data linked to each keyword broken down into keywords such as "people" and "vehicles".

[0103] Furthermore, when it is detected that "animals" has been uttered, the visual communication system 4 updates and displays the text information LI2 of "people and vehicles and animals" and the 2D thumbnail data SM21-SM28 linked thereto in the thumbnail display area S300, as shown in Fig. 18. That is, every time the visual communication system 4 detects the input of linguistic information, it updates and displays the 2D thumbnail data in real time. Note that the thumbnail data linked to the text information may be thumbnail data linked to the entire text information of "people and vehicles and animals", or it may be a group of thumbnail data linked to each keyword broken down into keywords such as "people", "vehicles", and "animals".

[0104] 17 and 18, the visual communication system 4 can quickly update and display new thumbnail images on the display screen every time language information is received from the user. This allows the user to select one or more thumbnail images that are closest to the scene in his or her mind when speaking or inputting text, without stopping the conversation.

[0105] When it is detected that the 2D thumbnail data SM14 has been selected on the screen of FIG. 17, the visual communication system 4 calls up the 3D illustration data IL1 linked to the 2D thumbnail data SM14 and displays it in the drawing result display area S100, as shown in FIG. 19.

[0106] In the three-dimensional editing process, the visual communication system 4 accepts three-dimensional rotation operations, three-dimensional movement operations, three-dimensional enlargement operations, three-dimensional reduction operations, etc., and in response to these operation requests, changes the display form of the three-dimensional illustration data IL1 three-dimensionally, as shown in Figure 20.

[0107] When it is detected that any of the buttons for two-dimensional editing (the add button S2013, the color button S2014, the eraser button S2015) has been pressed, the visual communication system 4 fixes the three-dimensional illustration data IL1 to two-dimensional illustration data IL2, as shown in Fig. 21. In this embodiment, as an example, pressing of the button for two-dimensional editing is determined to be acceptance of a request to turn off the language recognition function, and the system transitions to a state in which two-dimensional editing is possible.

[0108] In the two-dimensional editing process, the visual communication system 4 accepts two-dimensional rotation operations, two-dimensional movement operations, two-dimensional enlargement operations, two-dimensional reduction operations, partial erasure operations, operations to add hand-drawn line drawings, coloring operations, and the like, and in response to these operation requests, changes the display form of the two-dimensional illustration data IL2 two-dimensionally as shown in Figures 21, 22, and 23.

[0109] In FIG. 21, the retouch button S2013 is used to add handwritten lines of scenery that will form the background of the two-dimensional illustration data IL2, and the color button S2014 is used to add color to the three-dimensional illustration data IL2.

[0110] Also, in Fig. 21, the language information recognition status display icon, character information, and thumbnail data that were displayed in the thumbnail display area S300 in Fig. 19 and Fig. 20 are not displayed. The timing for hiding the various information in the thumbnail display area S300 can be appropriately selected, such as the timing when it is detected that any of the buttons for two-dimensional editing processing (the Add button S2013, the Color button S2014, the Eraser button S2015) has been pressed, the timing when fixing is executed, the timing when two-dimensional editing processing is started by the user, or the like.

[0111] Furthermore, even after performing two-dimensional editing, by pressing the voice input button S2011 or the key input button S2012, the language information recognition function can be turned on and various information can be displayed in the thumbnail display area S300, allowing new three-dimensional illustration data to be displayed and edited on the image that has been two-dimensionally edited, and new fixed two-dimensional illustration data to be added.

[0112] In FIG. 22, part of the line drawing in the two-dimensional illustration data IL2' is deleted by pressing the eraser button S2015.

[0113] In FIG. 23, a line drawing is added to the two-dimensional illustration data IL2'' by clicking the add button S2013.

[0114] As illustrated in Figures 19 to 23, the first stage editing process (Figures 19 and 20) and the second stage editing process (Figures 21 to 23) make it possible to accurately express, in a short amount of time, an illustration that closely resembles the scene that occurs in the user's mind when the speech or text is input.

[0115] As described above, in this embodiment, the visual communication system 4 searches for and displays an illustration corresponding to the linguistic information received from the user, and enables a two-stage editing process while changing the display form of the displayed illustration. This can support accurate information transmission in communication situations.

[0116] The concept of the visual communication system 4 may be applied to an online conference system 24. The online conference system 24 may be configured as shown in Fig. 24. Fig. 24 is a diagram showing the configuration of the online conference system 24 according to a modified example of the embodiment, and shows an example in which an online conference tool is used as the drawing display unit 400.

[0117] The online conference system 24 includes a plurality of client terminals 21a, 21b, a communication management server 22, and a connection unit 23. The connection unit 23 connects the plurality of client terminals 21a, 21b and the communication management server 22 so that they can communicate with each other. The plurality of client terminals 21a, 21b can share a display screen realized by a drawing display unit 400.

[0118] Each of the client terminals 21a and 21b has a language information input unit 100a and 100b, an illustration display unit 200a and 200b, and a drawing operation unit 300a and 300b. The functions and operations of the language information input unit 100a and 100b, the illustration display unit 200a and 200b, and the drawing operation unit 300a and 300b are similar to the functions and operations of the language information input unit 100, the illustration display unit 200, and the drawing operation unit 300 in the embodiment, respectively.

[0119] The communication management server 22 includes a communication management unit 600 in addition to a drawing display unit 400 and a storage unit 500. The functions and operations of the drawing display unit 400 and the storage unit 500 are similar to those of the drawing display unit 400 and the storage unit 500 in the embodiment, respectively.

[0120] The communication management unit 600 transmits and manages audio and camera images received from each of the multiple client terminals 21a and 21b, which are conference participants, to the other client terminals. The drawing display unit 400 draws and displays the images by superimposing them on the camera images of the online conference. The communication management unit 600 transmits the superimposed images to the other client terminals.

[0121] The superimposition may be performed on a part of the image or on the entire image. In addition to the drawn image, S200, S300, etc. may also be superimposed at the same time.

[0122] For example, as shown in Figs. 25 to 27, a camera image showing a user and a drawing made by a gesture of the user acquired by a motion acquisition device can be superimposed. Fig. 25 is a diagram showing a camera image captured by the camera 520 of the client terminal 21a in a modified example of the embodiment, and illustrates a camera image showing the user of the client terminal 21a. Fig. 26 is a diagram showing an image superimposed on the camera image of Fig. 25 by the drawing display unit 400 in a modified example of the embodiment. Fig. 26 shows an example in which the drawing result display area S100 and the thumbnail display area S300 are superimposed, but the operation palette S200 is not superimposed, among the drawing result display area S100, the operation palette S200, and the thumbnail display area S300. Fig. 27 is a diagram showing an image displayed on another client terminal 21b in a modified example of the embodiment. In this case, handwriting by a user with a fingertip on his / her display at a certain client terminal is displayed as an image in which a drawing is superimposed on the camera image by the user's fingertip at the other client terminal.

[0123] In this way, visual communication using illustrations can be achieved through gesture manipulation while talking.

[0124] In the above-described embodiment, the visual communication system 4 and the online conference system 24 are examples of an image editing system or an image display system. The client terminal 1, the client terminal 21a, the client terminal 21b, the server 2, and the communication management terminal 22 are examples of an image editing device or an image display device.

[0125] Each function of the above-described embodiments can be realized by one or more processing circuits. In this specification, the term "processing circuit" includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to execute each function described above.

[0126] The correspondence table in the specification may be generated by the learning effect of machine learning. Here, machine learning is a technology for enabling a computer to acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates an algorithm required for judgments such as data identification from learning data previously loaded, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. Any learning method for machine learning is acceptable.

[0127] The program 500a executed by the visual communication system 4 or the online conference system 24 may be provided by being pre-installed in a ROM or the like. Alternatively, the program 500a may be provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD) in the form of an installable or executable file. Alternatively, the program 500a may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program 500a may also be provided or distributed via a network such as the Internet. [Explanation of symbols]

[0128] 1. Client terminal 2 Server 3 Connection 4. Visual Communication Systems 21a, 21b Client terminal 22 Communication Management Server 23 Connection 24 Online conference system [Prior art documents] [Patent documents]

[0129] [Patent Document 1] Patent No. 6339529

Claims

1. A first input means for receiving voice input information, text input information, or action input information; a recognition means for recognizing character information from the voice input information, the text input information, or the action input information; a search means for searching for a plurality of thumbnail images each associated with image data by using the recognized character information; a display means for displaying the retrieved thumbnail images; a first editing means for performing a first editing process on first image data that is associated with a thumbnail image selected from the plurality of thumbnail images displayed on the display means and has a first data format; a second editing means for performing a second editing process on second image data having a second data format different from the first data format among the image data displayed on the display means; a conversion means for converting the first data format into the second data format, The display means is when the first editing process is performed by the first editing means, a display of the first image data is changed to a display of image data after the first editing process; when the second editing process is performed on the second image data converted by the conversion means, a display of the second image data is changed to a display after the second editing process. Image editing device.

2. When none of the plurality of thumbnail images displayed on the display means is selected and the first input means receives second voice input information, second text input information or second action input information and second character information is recognized by the recognition means for the second voice input information, the second text input information or the second action input information, the search means uses the recognized second character information to search for a plurality of second thumbnail images each associated with image data, and the display means displays the searched second thumbnail image in place of the plurality of thumbnail images displayed on the display means. The image editing device according to claim 1 .

3. A second input means for receiving a selection operation for designating a thumbnail image to be selected from the plurality of thumbnail images; a selection means for selecting image data linked to a thumbnail image designated by the selection operation from among a plurality of image data linked to a plurality of thumbnail images displayed on the display means; Further equipped with The first editing means performs the first editing process on the first image data when the first image data is selected by the selection means from among a plurality of image data linked to a plurality of thumbnail images displayed on the display means. The image editing device according to claim 1 .

4. The conversion means converts the first data format into the second data format in response to the second editing process being enabled by the second editing means. The image editing device according to claim 1 .

5. the first editing means performs the first editing process on the three-dimensional image data displayed on the display means; the second editing means performs the second editing process on the two-dimensional image data displayed on the display means; The conversion means converts a three-dimensional image data format into a two-dimensional image data format. The image editing device according to claim 1 .

6. The conversion means obtains a plurality of planar coordinates projected onto a plane corresponding to the display means from a plurality of spatial coordinates included in the three-dimensional image data, and generates two-dimensional image data including the plurality of planar coordinates. The image editing device according to claim 5.

7. the first editing process includes a process corresponding to a three-dimensional editing operation; The second editing process includes a process corresponding to a two-dimensional editing operation. The image editing device according to claim 1 .

8. A step of receiving voice information or text input information; recognizing character information from the voice information or the text input information; using the recognized character information to search for a plurality of thumbnail images, each of which is associated with an image data; displaying the retrieved thumbnail images on a display means; a step of performing a first editing process on first image data that is associated with a thumbnail image selected from the plurality of thumbnail images displayed on the display means and has a first data format; a step of changing a display of the first image data to a display of image data after the first edit processing when the first edit processing is performed in the step of performing the first edit processing; converting the first data format into a second data format; a step of performing a second editing process on second image data having the second data format among the image data displayed on the display means; a step of changing a display of the second image data to a display after the second editing process when the second editing process is applied to the second image data converted in the converting step in a step of applying the second editing process; Image editing methods including.

9. A step of receiving voice information or text input information; recognizing character information from the voice information or the text input information; using the recognized character information to search for a plurality of thumbnail images, each of which is associated with an image data; displaying the retrieved thumbnail images on a display means; a step of performing a first editing process on first image data that is associated with a thumbnail image selected from the plurality of thumbnail images displayed on the display means and has a first data format; a step of changing a display of the first image data to a display of image data after the first edit processing when the first edit processing is performed in the step of performing the first edit processing; converting the first data format into a second data format; a step of performing a second editing process on second image data having the second data format among the image data displayed on the display means; a step of changing a display of the second image data to a display after the second editing process when the second editing process is applied to the second image data converted in the converting step in a step of applying the second editing process; An image editing program that causes a computer to execute the following:

Images