Information processing device, input / output device, information processing method, method, program, model, and information processing system

Abstract

To provide an information processing device, information processing system, and method for generating text information for a display screen in line with a user's intention.SOLUTION: An information processing system includes: a terminal device 10 that is a glass device or a wearable device and an input / output device; and a server 40 that is an information processing device connected to the terminal device via a network 100. The server includes: a storage unit 4000 configured to store a tacit knowledge model 4004 generated by executing learning processing with images and texts as learning data; and a text generation unit 45 configured to generate text information based on a target image and the tacit knowledge model 4004, the target image being identified by an identification unit 44 from a display screen 1000 displayed on a display.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, an input / output device, an information processing method, a method, a program, a model, and an information processing system. [Background technology]

[0002] Patent Document 1 discloses a technique for extracting a matching object to be matched with 3D model data from point cloud data consisting of depth information, matching the 3D model data with the matching object, and identifying object matching points in the 3D model data using machine learning. Patent Document 1 also discloses a technique for identifying the position of an object within a building based on information on the object matching points in the 3D model data and depth information of the point cloud data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7113611 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to generate text information for a display screen in accordance with a user's intention. [Means for solving the problem]

[0005] The information processing device of the present invention comprises a storage means for storing a model generated by executing a learning process using images and text as learning data, and a text information generation means for generating text information based on an identified target image on a display screen displayed on a display unit and the model. Effect of the Invention

[0006] According to the present invention, text information for a display screen that matches the user's intention can be generated. [Brief description of the drawings]

[0007] [Figure 1] 1 is an overall configuration diagram of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a hardware configuration diagram of a terminal device and a server according to the embodiment. [Diagram 3] FIG. 1 is a functional block diagram of an information processing system according to an embodiment of the present invention. [Figure 4] 4A and 4B are conceptual diagrams illustrating an example of a user information management table and an image management table according to the embodiment. [Diagram 5] FIG. 11 is a sequence diagram showing an example of a text information generation process according to the embodiment. [Figure 6] FIG. 11 is a sequence diagram illustrating an example of a model update process according to the embodiment. [Figure 7] 5 is a flowchart illustrating an example of various processes according to the embodiment. [Figure 8] 11 is an explanatory diagram of a model update process and a text information generation process according to the embodiment. FIG. [Figure 9] 11 is another explanatory diagram of the model update process and the text information generation process according to the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] In industries such as civil engineering and construction, the shift to BIM / CIM is being promoted in order to address issues such as the declining birthrate and aging population and to improve labor productivity.

[0009] BIM is an abbreviation for Building Information Modeling, and is a solution that uses a building database that adds attribute data such as cost, finish, and management information to a three-dimensional digital model of a building (hereafter referred to as a 3D model) created on a computer, allowing information to be utilized in every process from building design and construction to maintenance.

[0010] CIM is an abbreviation for Construction Information Modeling, and is a solution for the civil engineering field (general infrastructure such as roads, electricity, gas, and water) proposed following the example of BIM, which was being promoted in the construction field. Like BIM, it is being worked on as a way to improve the efficiency and sophistication of the entire construction production system by sharing information between related parties, mainly using 3D models.

[0011] The key to promoting BIM / CIM is how to utilize the BIM / CIM that has been created.

[0012] Specifically, the 3D space restored by BIM / CIM can be used for design and construction purposes, as well as other tasks such as maintenance and site inspections. In other words, in addition to using it as a blueprint, other uses such as keeping records in 3D space and sharing it with others can be considered.

[0013] In addition, because digital work can be recorded as logs, if tacit knowledge can be extracted from these, it can be effectively used for transferring skills from experienced workers to younger workers. This is expected to lead to front-loading of work and human resource development.

[0014] Here, when we focus on the transfer of tacit knowledge, as mentioned above, it can be said that the transfer of tacit knowledge between different works and between users with different levels of expertise is an issue not only in 3D space but also in spherical images, flat images, and the like.

[0015] In view of the above, an object of this embodiment is to transfer tacit knowledge about images between different works or between users with different levels of proficiency.

[0016] 1 is a diagram showing an overall configuration of an information processing system according to an embodiment of the present invention. The information processing system 1 of the present embodiment is constructed by a terminal device 10, which is an example of an input / output device, and a server 40.

[0017] The server 40 is an example of an information processing device connected to an input / output device via a network. The terminal device 10 may be a glass device or a wearable device, and the information processing system 1 may include a plurality of terminal devices.

[0018] The terminal device 10 and the server 40 can communicate with each other via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), and the like. The communication network 100 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution). The terminal device 10 can communicate with each other using a short-range communication technology such as NFC (Near Field Communication) (registered trademark).

[0019] 2 is a diagram showing the hardware configuration of the diagnostic device and management device according to this embodiment. The hardware configuration of the terminal device 10 is the same, and is indicated by a reference number in the 100s as the terminal device 10. The hardware configuration of the management device is indicated by a reference number in the 400s.

[0020] The hardware configuration of the terminal device 10 will be described below, but the hardware configuration of the server 40 is similar and therefore will not be described.

[0021] The terminal device 10 is constructed by a computer and, as shown in FIG. 2, includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a HD (Hard Disk) 104, a HDD (Hard Disk Drive) controller 105, a display I / F 106 and a communication I / F 107.

[0022] Of these, the CPU 101 controls the overall operation of the terminal device 10. The ROM 102 stores programs such as an IPL (Initial Program Loader) used to drive the CPU 101. The RAM 103 is used as a work area for the CPU 101.

[0023] The HD 104 stores various data such as programs, etc. The HDD controller 105 controls reading and writing of various data from and to the HD 10 under the control of the CPU 101.

[0024] The display I / F 106 is a circuit that displays images on the display 106a. The display 106a is a type of display unit such as a liquid crystal or organic EL (Electro Luminescence) that displays various information such as a cursor, a menu, a window, characters, or an image. The communication I / F 107 is an interface used for communication with other devices.

[0025] When the terminal device 10 is a glass device, the terminal device 10 may use, instead of the display I / F 106, a circuit for displaying an image on a lens or the like serving as a transmissive / reflective member.

[0026] The communication I / F 107 is, for example, a network interface card (NIC) compatible with a transmission control protocol (TCP) / internet protocol (IP).

[0027] The terminal device 10 also includes a sensor I / F 108 , a sound input / output I / F 109 , an input I / F 110 , a media I / F 111 , and a DVD-RW (Digital Versatile Disk Rewritable) drive 112 .

[0028] The sensor I / F 108 is an interface that receives detection information from various sensors. The audio input / output I / F 109 is a circuit that processes input and output of audio signals between the speaker 109a and the microphone 109b under the control of the CPU 101. The input I / F 110 is an interface for connecting a predetermined input means to the terminal device 10.

[0029] The keyboard 110a is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The mouse 110b is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, operating the display screen, etc.

[0030] The media I / F 111 controls reading and writing (storing) of data from and to a recording medium 111a such as a flash memory. The DVD-RW drive 112 controls reading and writing of various data from and to a DVD-RW 112a, which is an example of a removable recording medium. Note that the medium is not limited to DVD-RW, and may be a DVD-R or the like. The DVD-RW drive 112 may also be a Blu-ray drive that controls reading and writing of various data from and to a Blu-ray Disc (registered trademark).

[0031] The terminal device 10 also includes a bus line 113. The bus line 113 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 101.

[0032] In addition, recording media such as HDs and CD-ROMs storing the above programs can be provided domestically or internationally as program products. The terminal device 10, for example, realizes the information processing method according to the present invention by executing the program according to the present invention.

[0033] FIG. 3 is a functional block diagram of the information processing system according to the present embodiment.

[0034] As shown in Fig. 3, the terminal device 10 has a transmission / reception unit 11, an input reception unit 12, a display control unit 13, an audio control unit 14, a conversion unit 15, and a storage / readout unit 19. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 101 in accordance with a program loaded from the HD 104 onto the RAM 103. The terminal device 10 also has a storage unit 2000 constructed by the RAM 103 and the HD 104 shown in Fig. 2.

[0035] (Functional configuration of terminal device) Next, each component of the terminal device 10 will be described.

[0036] The transmission / reception unit 11 is an example of a transmission means, and is realized by instructions from the CPU 101 shown in FIG. 2 and the communication I / F 107, and transmits and receives various data (or information) with other terminals, devices or systems via the communication network 100.

[0037] The input receiving unit 12 is an example of an input receiving means, and is mainly realized by instructions from the CPU 101 shown in FIG. 2, as well as the input I / F 110 and the sound input / output I / F 109, and receives various inputs from the user using the microphone 109b, keyboard 110a and mouse 110b.

[0038] The display control unit 13 is an example of a display control means and output means, and is realized by instructions from the CPU 101 shown in FIG. 2 and the display I / F 106, and causes various images and screens to be displayed on the display 106a, which is an example of a display unit.

[0039] When the terminal device 10 is a glass device, the display control unit 13 causes a transmissive reflective member such as a lens to display a virtual image instead of the display I / F 106.

[0040] The audio control unit 14 is an example of an audio control means and output means, and is realized by instructions from the CPU 101 shown in FIG. 2 and the audio input / output I / F 109, and causes the speaker 109a, which is an example of an audio playback unit, to play back sound.

[0041] The conversion unit 15 is an example of a processing means, and is realized by instructions from the CPU 101 shown in FIG. 2, and performs a process of converting character information into voice information and a process of converting voice information into character information.

[0042] The memory / read unit 19 is an example of a memory control means, and is executed by instructions from the CPU 101 shown in FIG. 2, as well as the HD 104, the media I / F 111, and the DVD-RW drive 112, to store various data in the memory unit 2000, the recording medium 111a, and the DVD-RW 112a, and to read various data from the memory unit 2000, the recording medium 111a, and the DVD-RW 112a.

[0043] <Server functional configuration> The server 40 has a transmission / reception unit 41, a screen generation unit 42, a judgment unit 43, a specification unit 44, a text generation unit 45, an update unit 46, and a storage / readout unit 49. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 401 in accordance with a program loaded from the HD 404 onto the RAM 403. The server 40 also has a memory unit 4000 constructed by the HD 404 shown in Fig. 2. The memory unit 4000 is an example of a memory means.

[0044] (Management server functional configuration) Next, each component of the server 40 will be described. The server 40 may be configured to distribute and realize each function among multiple computers. Furthermore, the server 40 will be described as a server computer that exists in a cloud environment, but may be a server that exists in an on-premise environment.

[0045] The transmission / reception unit 41 is an example of a transmission means, and is realized by instructions from the CPU 401 shown in FIG. 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices or systems via the communication network 100.

[0046] Screen generation unit 42 is an example of a screen generation means, and is realized by instructions from CPU 401 shown in FIG. 2, and generates various screens.

[0047] The determination unit 43 is an example of a determination means, and is realized by instructions from the CPU 401 shown in FIG. 2, and performs various determinations described below.

[0048] The specification unit 44 is an example of a specification means, and is realized by an instruction from the CPU 401 shown in FIG. 2, and specifies the target image.

[0049] Text generation unit 45 is an example of a text information generating means, and is realized by an instruction from CPU 401 shown in FIG. 2, to generate text information.

[0050] The update unit 46 is an example of an update means, and is realized by instructions from the CPU 401 shown in FIG. 2, and performs various settings and decisions, which will be described later.

[0051] The storage / reading unit 49 is an example of a storage control means, and is executed by commands from the CPU 401 shown in Fig. 2, as well as the HDD 404, the media I / F 411, and the DVD-RW drive 412, to store various data in the storage unit 4000, the recording medium 411a, and the DVD-RW 412a, and to read various data from the storage unit 4000, the recording medium 411a, and the DVD-RW 412a. The storage unit 4000, the recording medium 411a, and the DVD-RW 412a are examples of storage means.

[0052] In the storage unit 4000, a user information management DB 4001 constituted by a user information management table, an image management DB 4002 constituted by an image information management table, a caption model 4003, a tacit knowledge model 4004, and a large-scale language model 4005 are constructed.

[0053] The user information management DB 4001 stores and manages various types of information, and the image management DB 4002 stores and manages various types of images such as three-dimensional images related to three-dimensional point groups and three-dimensional models, and spherically captured images.

[0054] The caption model 4003 is a model that is generated by executing a learning process using a combination of an image and a caption comment as learning data, and causes a computer to function to output a caption comment based on the image.

[0055] Here, the caption comment is text data, and is a comment that describes an image, among comments expressed by voice or characters.

[0056] The tacit knowledge model 4004 is a model that is generated by executing a learning process using a combination of an image and a tacit knowledge comment for the image as learning data, and causes a computer to function to output a tacit knowledge comment based on the image.

[0057] Here, the tacit knowledge comment is text data, and is a comment expressed by voice or text, excluding caption comments, that is, a comment relating to content not expressed in an image.

[0058] The large-scale language model 4005 is a computer language model that is generated by performing a learning process using a huge amount of unlabeled text as learning data, and is composed of an artificial neural network having a large number of parameters.

[0059] The large-scale language model 4005 can capture much of the syntax and meaning of human language by being sufficiently trained using techniques for learning context, such as next sentence prediction, which understands the context by determining whether sentence 1 and sentence 2 are consecutive, or a masked language model, which understands the context by masking words in a sentence and predicting the masked words from the words before and after them.

[0060] FIG. 4 is a conceptual diagram showing an example of a user information management table according to the present embodiment.

[0061] A user information management DB 4001 is constructed in the storage unit 4000, which is configured by a user information management table as shown in Fig. 4. In this user information management table, attribute information indicating the attributes of a user and the presence or absence of the authority to use the text generating unit 45 are associated and managed for each user ID. The attribute information includes information such as manager, worker, contractor, skilled worker, and unskilled worker.

[0062] FIG. 5 is a conceptual diagram showing an example of an image management table according to the present embodiment.

[0063] In the storage unit 4000, an image management DB 4002 is created, which is made up of an image management table as shown in FIG.

[0064] In this image management table, for each image ID, image information such as a three-dimensional image or a spherical image, acquisition information identifying the acquisition scene in which the image information was acquired, and usage information identifying the usage scene in which the image information or a display screen including the image information is used are associated and managed.

[0065] The acquisition information includes the date on which the image information was acquired, the name of the property indicated by the image information, the name of the process in which the image information was acquired, etc., and the usage information includes the date on which the image information, etc. was used, the name of the property in which the image information, etc. was used, the name of the process in which the image information, etc. was used, and multiple pieces of usage information are managed in an image management table.

[0066] FIG. 5 is a sequence diagram showing an example of the text information generation process according to the present embodiment.

[0067] The input reception unit 12 of the terminal device 10 receives an input operation related to the user ID, image ID, and usage information of the user on the input / output screen displayed on the display 106a (step S1). The transmission / reception unit 11 transmits the user ID, image ID, and usage information received in step S1 to the server 40, and the transmission / reception unit 41 of the server 40 receives the user ID, image ID, and usage information transmitted from the terminal device 10 (step S2).

[0068] Next, the storage / reading unit 49 of the server 40 stores the usage information received in step S2 in the image management DB 4002 in association with the image ID, and searches the image management DB 4002 using the image ID received in step S2 as a search key to read out the image information, acquisition information and usage information corresponding to the image ID (step S3).

[0069] Screen generation unit 42 generates a display screen including the image information read out by storage / readout unit 49 (step S4).

[0070] Here, in step S1, a plurality of image IDs are input, and in step S3, the image management DB 4002 is searched using the plurality of image IDs as search keys, thereby making it possible to read out information about a plurality of images.

[0071] Also, in step S1, instead of the image ID, a material ID for identifying a material including a desired image may be input, and the image management DB 4002 may manage one or more pieces of image information in association with each material ID.

[0072] As a result, in step S2, the transmission / reception unit 11 transmits the material ID instead of the image ID, and in step S3, one or more pieces of image information corresponding to the material ID can be read out by searching the image management DB 4002 using the material ID as a search key.

[0073] The transmitting / receiving unit 41 transmits display screen information related to the display screen generated in step S4 to the terminal device 10, and the transmitting / receiving unit 11 of the terminal device 10 receives the display screen information transmitted from the server 40 (step S5). As another embodiment, instead of steps S1 to S5, the transmitting / receiving unit 11 of the terminal device 10 may receive information indicating a captured image transmitted from an imaging device as the display screen information.

[0074] The display control unit 13 of the terminal device 10 causes the display 106a to display the display screen received in step S5 (step S6). As another embodiment, the display control unit 13 may cause the display 106a to display an image managed by the terminal device 10 as a display screen, instead of the display screen received in step S5. The input accepting unit 12 of the terminal device 10 accepts input information input by the user to the displayed display screen (step S7).

[0075] This input information includes voice information, character information, and operation information input by the user, and the conversion unit 15 may convert the voice information included in the input information into character information. The voice information, character information, and operation information include specific information for identifying a target image on the display screen, and a question for the target image.

[0076] The transmitting / receiving unit 11 transmits the input information accepted by the input accepting unit 12 to the server 40, and the transmitting / receiving unit 41 of the server 40 receives the input information transmitted from the terminal device 10 (step S8). As another embodiment, the transmitting / receiving unit 11 may transmit display screen information, in which the captured image or the image managed by the terminal device 10 is displayed as a display screen, to the server 40 together with the input information, and the transmitting / receiving unit 41 of the server 40 may receive the input information transmitted from the terminal device 10 and the display screen information.

[0077] The memory / read unit 49 searches the user information management DB 4001 using the user ID received in step S2 as a search key to read out the user attributes and usage rights associated with the user ID, and the judgment unit 43 judges whether the user has usage rights for the text generation unit 45 based on the usage rights read from the user information management DB 4001 (step S9).

[0078] If it is determined in step S9 that the user has the usage authority, the specification unit 44 specifies the target image on the display screen based on the specification information included in the input information received in step S8 (step S10).

[0079] The text generation unit 45 acquires tacit knowledge comments based on the tacit knowledge model 4004 using the target image identified in step S10, and generates text information based on the large-scale language model 4005 using the tacit knowledge comments and the questions extracted from the input information received in step S8 (step S11).

[0080] The text generating unit 45 may convert voice information contained in the input information into character information, and the text information generated by the text generating unit 45 may be either voice information or character information.

[0081] The text generating unit 45 may generate text information without using a question, or may generate fixed questions in the system and use the fixed questions. In this case, the questions are not visible to the user. Alternatively, the text generating unit 45 may generate fixed questions in the system, display the fixed questions on the display unit and allow the user to select one, and use the selected question.

[0082] The transmitting / receiving unit 41 transmits the text information generated in step S11 to the terminal device 10, and the transmitting / receiving unit 11 of the terminal device 10 receives the text information transmitted from the server 40 (step S12).

[0083] If the received text information is character information, the display control unit 13 of the terminal device 10 causes the display 106a to display the text information, or the conversion unit 15 converts the received text information into voice information, and the voice control unit 14 causes the speaker 109a to reproduce the converted text information (step S13). Also, if the received text information is voice information, the speaker 109a reproduces the text information, or the conversion unit 15 converts the received text information into character information, and the display 106a displays the converted text information.

[0084] In the above, if the information processing system 1 has multiple terminal devices 10, steps S1 and S2, steps S5 and S6, steps S7 and S8, and steps S12 and S13 may be performed by different terminal devices, or may be performed by multiple terminal devices.

[0085] 3 may be integrated into the terminal device 10, and the processing of the server 40 in FIG.

[0086] FIG. 6 is a sequence diagram showing an example of a model update process according to this embodiment.

[0087] The input reception unit 12 of the terminal device 10 receives an input operation related to the user's user ID, image ID, and usage information on the input / output screen displayed on the display 106a (step S21). The transmission / reception unit 11 transmits the user ID, image ID, and usage information received in step S21 to the server 40, and the transmission / reception unit 41 of the server 40 receives the user ID, image ID, and usage information transmitted from the terminal device 10 (step S22).

[0088] Next, the storage / readout unit 49 of the server 40 stores the usage information received in step S22 in the image management DB 4002 in association with the image ID, and searches the image management DB 4002 using the image ID received in step S22 as a search key to read out the image information, acquisition information, and usage information corresponding to the image ID (step S23). Screen generation unit 42 generates a display screen including the image information read out by storage / readout unit 49 (step S24).

[0089] Here, in step S21, a plurality of image IDs are input, and in step S23, image management DB 4002 is searched using the plurality of image IDs as search keys, thereby making it possible to read out information about a plurality of images.

[0090] Also, in step S21, instead of the image ID, a material ID for identifying a material including a desired image may be input, and the image management DB 4002 may manage one or more pieces of image information in association with each material ID.

[0091] As a result, in step S22, the transmission / reception unit 11 transmits the material ID instead of the image ID, and in step S23, the image management DB 4002 is searched using the material ID as a search key, thereby reading out one or more pieces of image information corresponding to the material ID.

[0092] The transmitting / receiving unit 41 transmits display screen information relating to the display screen generated in step S24 to the terminal device 10, and the transmitting / receiving unit 11 of the terminal device 10 receives the display screen information transmitted from the server 40 (step S25). As another embodiment, instead of steps S21 to S25, the transmitting / receiving unit 11 of the terminal device 10 may receive information indicating a captured image transmitted from an imaging device as the display screen information.

[0093] The display control unit 13 of the terminal device 10 causes the display 106a to display the display screen received in step S25 (step S26). As another embodiment, the display control unit 13 may cause the display 106a to display an image managed by the terminal device 10 as a display screen, instead of the display screen received in step S25. The input accepting unit 12 of the terminal device 10 accepts input information input by the user to the displayed display screen (step S27).

[0094] The input information includes voice information, character information, and operation information input by the user, and the conversion unit 15 converts the voice information included in the input information into character information. The voice information, character information, and operation information include specific information that identifies a target image on the display screen. The voice information and character information also include a caption comment that explains the target image, and a tacit knowledge comment related to content not expressed in the target image.

[0095] The transmitting / receiving unit 11 transmits input information related to the input operation accepted by the input accepting unit 12 to the server 40, and the transmitting / receiving unit 41 of the server 40 receives the input information transmitted from the terminal device 10 (step S28). As another form, the transmitting / receiving unit 11 may transmit display screen information in which the captured image or the image managed by the terminal device 10 is displayed as a display screen together with the input information to the server 40, and the transmitting / receiving unit 41 of the server 40 may receive the input information transmitted from the terminal device 10 and the display screen information.

[0096] The identification unit 44 identifies a target image on the display screen based on the identification information included in the input information received in step S28 (step S29).

[0097] The judgment unit 43 acquires a caption comment based on the caption model 4003 using the target image identified in step S29, and judges the relevance between the caption comment and the comment included in the input information received in step S28 (step S30).

[0098] The judgment unit 43 may judge the relevance of all comments included in the input information received in step S28 with the acquired caption comment, or may divide the comments included in the input information received in step S28 into multiple comments and judge the relevance of each of the divided comments with the acquired caption comment.

[0099] The update unit 46 updates the caption model 4003 by treating the comments determined to have a high relevance in step S30 as caption comments and by using them as learning data together with the target image identified in step S29, and updates the tacit knowledge model 4004 by treating the comments determined to have a low relevance in step S30 as tacit knowledge comments and by using them as learning data together with the target image identified in step S29 (step S31).

[0100] In step S31, the storage / read unit 49 searches the user information management DB 4001 using the user ID received in step S22 as a search key to read out user attributes associated with the user ID, and the update unit 46 updates the tacit knowledge model 4004 as learning data by including the user attributes, and the acquired information and usage information read out in step S23 related to the target image in the tacit knowledge comment.

[0101] In the above, when the information processing system 1 has multiple terminal devices 10, steps S21 and S22, steps S25 and S26, and steps S27 and S28 may be executed by different terminal devices, or may be executed by multiple terminal devices, and the terminal device 10 shown in FIG. 6 may be different from the terminal device shown in FIG. 5.

[0102] 3 may be integrated into the terminal device 10, and the processing of the server 40 in FIG. 6 may be executed by the terminal device 10. In addition, the functions of the server 40 in FIG.

[0103] FIG. 7 is a flowchart showing an example of various processes according to the present embodiment.

[0104] FIG. 7(a) shows a process corresponding to step S10 in FIG. 5 and step S29 in FIG.

[0105] The judgment unit 43 judges whether the operation information included in the input information received from the terminal device 10 includes an operation to identify a target image (step S41), and if the operation to identify is included, the identification unit 44 identifies the target image according to the identification operation (step S42).

[0106] The determination unit 43 determines whether the voice information or text information included in the input information received from the terminal device 10 includes a comment that identifies the target image (step S43), and if an identifying comment is included, the identification unit 44 identifies the target image according to the identifying comment (step S44). The comment that identifies the target image is, for example, a comment that identifies a position on the display screen, such as right or left, or a comment that identifies acquisition information of an image.

[0107] If the operation information does not include an operation for identifying the target image, and the voice information and the text information do not include a comment for identifying the target image, the identifying unit 44 identifies the entire display screen as the target image (step S45).

[0108] In step S42, when the operation information includes an operation for identifying each of the plurality of target images, the identification unit 44 may identify each of the plurality of target images in response to each operation.

[0109] In step S44, when the voice information or text information includes a comment that identifies each of the multiple target images, the identification unit 44 may identify each of the multiple target images according to each comment.

[0110] In addition, when the operation information includes an operation for identifying a target image and the audio information or text information includes a comment for identifying another target image, the identification unit 44 may identify the target image according to each of the operation and the comment.

[0111] Furthermore, in step S45, when the display screen includes text information, the specification unit 44 may specify a portion of the display screen excluding the text information as the target image.

[0112] FIG. 7(b) shows the process corresponding to steps S30 and S31 in FIG.

[0113] The determination unit 43 determines the relevance between the caption comment acquired from the caption model 4003 using the target image and the comment included in the input information received from the terminal device 10 (step S51). For example, when the rate at which the comment included in the input information matches the caption comment is equal to or greater than a predetermined value, the determination unit 43 determines that the relevance to the caption comment is high. Here, a high relevance to the caption comment means that there is a high probability that the comment describes the content of the target image.

[0114] The judgment unit 43 may judge the relevance of all comments included in the input information received from the terminal device 10 to the acquired caption comment, or may divide the comments included in the input information received from the terminal device 10 into multiple comments and judge the relevance of each of the divided comments to the acquired caption comment.

[0115] The judgment unit 43 may, for example, determine an object in a target image by image recognition or object detection, and then determine whether or not words related to a caption comment describing the object are included in the comments included in the input information, or the percentage of the number of words included, as the relevance.

[0116] The update unit 46 updates the tacit knowledge model 4004 by treating the comment determined to have low relevance in step S51 as a tacit knowledge comment, as learning data together with the target image, the acquired information and usage information related to the target image, and the user attributes (step S52).

[0117] The update unit 46 updates the caption model 4003 by treating the comment determined to have a high degree of relevance in step S51 as a caption comment and using it as learning data together with the target image (step S53).

[0118] Here, the tacit knowledge model 4004 and the caption model 4003 may be updated arbitrarily. That is, when a predetermined condition is satisfied, the update unit 46 may execute step S52 to update the tacit knowledge model 4004, or execute step S53 to update the caption model 4003.

[0119] FIG. 7(c) shows a process corresponding to step S11 in FIG.

[0120] The text generation unit 45 acquires tacit knowledge comments based on the tacit knowledge model 4004 using the target image (step 61), and generates text information based on the large-scale language model 4005 using the tacit knowledge comments, the question extracted from the input information received in step S8, the acquired information and usage information read out in step S3 related to the target image, and the user attributes read out in step S9 (step S62).

[0121] FIG. 8 is an explanatory diagram of the model update process and the text information generation process according to this embodiment.

[0122] FIG. 8(a) is an explanatory diagram of the model update process corresponding to steps S26 and S27 in FIG. 6 and FIGS. 7(a) and 7(b).

[0123] The display control unit 13 of the terminal device 10 causes the display screen 1000 received from the server 40 to be displayed on the display 106 a . The display screen 1000 includes an image 1100 and text 1200 .

[0124] The input receiving unit 12 of the terminal device 10 receives, as input information input by the user to the displayed display screen 1000, voice information indicating the dialogues Q1, A1, Q2, and A2 between the user M1 and the user M2 from the microphone 109b.

[0125] The specification unit 44 specifies the image 1100, which is the portion of the display screen 1000 excluding the text 1200, as the target image.

[0126] Then, the determination unit 43 determines the relevance between the caption comment acquired from the caption model 4003 using the target image 1100 and the dialogues Q1, A1, Q2, and A2.

[0127] The update unit 46 updates the tacit knowledge model 4004 by treating the comments determined to have a low relevance among the dialogues Q1, A1, Q2, and A2 as tacit knowledge comments and using them as learning data together with the target image 1100, etc., and updates the caption model 4003 by treating the comments determined to have a high relevance as caption comments and using them as learning data together with the target image 1100.

[0128] FIG. 8(b) is an explanatory diagram of the text information generation process corresponding to steps S6, S7, and S13 in FIG. 5 and FIGS. 7(a) and 7(c).

[0129] The display control unit 13 of the terminal device 10 causes the display screen 1000 received from the server 40 to be displayed on the display 106 a . The display screen 1000 includes an image 1110 and text 1210 .

[0130] The input receiving unit 12 of the terminal device 10 receives, via the microphone 109b, voice information indicating the questions Q11 and Q12 from the user M3 as input information input by the user to the displayed display screen.

[0131] The identification unit 44 identifies the image 1110, which is the portion of the display screen 1000 excluding the text 1210, as the target image.

[0132] The text generation unit 45 acquires tacit knowledge comments based on the tacit knowledge model 4004 using the target image 1110, and generates text information related to answers A11 and A12 to the questions Q11 and Q12, respectively, based on the large-scale language model 4005 using the tacit knowledge comments, questions Q11, Q12, etc.

[0133] The display control unit 13 of the terminal device 10 causes the text information relating to the answers A11 and A12 received from the server 40 to be displayed on the display 106a.

[0134] FIG. 9 is another explanatory diagram of the model update process and the text information generation process according to this embodiment.

[0135] FIG. 9(a) is an explanatory diagram of the model update process corresponding to steps S26 and S27 in FIG. 6 and FIGS. 7(a) and 7(b).

[0136] The display control unit 13 of the terminal device 10 causes the display 106a to display the display screen 1000 received from the server 40, and the display screen 1000 includes a first image 1100A and a second image 1100B.

[0137] The input receiving unit 12 of the terminal device 10 receives, as input information input by the user to the displayed display screen 1000, character information indicating the comments C1 to C4 by the user M4 from the keyboard 110a.

[0138] Furthermore, the input receiving unit 12 receives, as input information input by the user to the displayed display screen 1000, operation information indicating an operation by the user M4 to specify the partial image 1100B1 in the first image 1100B, from the mouse 110b.

[0139] The identification unit 44 identifies the first image 1100B in the display screen 1000 as the target image in response to the operation information. The identification unit 44 may identify the partial image 1100B1 as the target image.

[0140] Then, the determination unit 43 determines the degree of association between the caption comment acquired from the caption model 4003 using the target image 1100B and the comments C1 to C4.

[0141] The update unit 46 updates the tacit knowledge model 4004 by treating the comments determined to have a low relevance among the comments C1 to C4 as tacit knowledge comments and using them as learning data together with the target image 1100B, etc., and updates the caption model 4003 by treating the comments determined to have a high relevance as caption comments and using them as learning data together with the target image 1100B.

[0142] FIG. 8(b) is an explanatory diagram of the text information generation process corresponding to steps S6, S7, and S13 in FIG. 5 and FIGS. 7(a) and 7(c).

[0143] The display control unit 13 of the terminal device 10 causes the display screen 1000 received from the server 40 to be displayed on the display 106a. The display screen 1000 includes an image 1110.

[0144] User M5 does not make any input to the displayed display screen 1000, and the input receiving unit 12 does not receive any input information input by the user to the displayed display screen 1000. The identification unit 44 identifies an image 1110, which is the entire display screen 1000, as the target image.

[0145] When the user M5 performs an operation to specify a partial image such as 1100B1 in Fig. 9(a) on the display screen 1000B, the input receiving unit 12 receives operation information indicating the operation to specify the partial image from the mouse 110b as input information. In this case, the specifying unit 44 specifies the partial image on the display screen 1000 as a target image in accordance with the operation information.

[0146] The text generation unit 45 acquires tacit knowledge comments based on the tacit knowledge model 4004 using the target image 1110, and generates text information related to the comments C11 to C14 using the tacit knowledge comments, etc., based on the large-scale language model 4005. The text generation unit 45 may generate text information using preset fixed questions.

[0147] The display control unit 13 of the terminal device 10 causes the text information relating to the comments C11 to C14 received from the server 40 to be displayed on the display 106a.

[0148] ●Summary● [First aspect] As described above, the server 40 according to one embodiment of the present invention includes the storage unit 4000 that stores the tacit knowledge model 4004 generated by executing a learning process using images and text as learning data, and the text generation unit 45 that generates text information based on the target image identified by the identification unit 44 from the display screen 1000 displayed on the display 106a, and the tacit knowledge model 4004.

[0149] Here, the server 40 is an example of an information processing device, and the text generating unit 45 is an example of a text information generating means.

[0150] With the above configuration, the server 40 can generate text information related to tacit knowledge in line with the user's intention, based on the target image identified on the display screen 1000.

[0151] [Second aspect] In the first aspect, the text generating unit 45 generates text information based on user information that identifies a user.

[0152] As a result, the server 40 can decide whether or not to generate text information based on the usage authority corresponding to the user information, change the content of the text information based on the user attributes corresponding to the user information, or change the content of the tacit knowledge model 4004 for generating text information.

[0153] [Third aspect] In the first or second aspect, the display screen 1000 is associated with at least one of acquisition information identifying an acquisition scene in which the image used to generate the display screen 1000 was acquired, and usage information identifying a usage scene in which the display screen 1000 is used.

[0154] This enables the server 40 to generate text information in line with the user's intentions based on the display screen 1000 based on images captured in various capture scenes and on target images on the display screen 1000 used in various usage scenes.

[0155] [Fourth aspect] In any of the first to third aspects, the text generation unit 45 generates text information based on at least one of acquisition information that identifies the acquisition scene in which the image that is the source of the display screen 1000 was acquired, and usage information that identifies the usage scene in which the display screen 1000 is used.

[0156] This allows the server 40 to generate text information that is more in line with the user's intentions depending on the acquisition scene and the usage scene.

[0157] [Fifth aspect] In any of the first to fourth aspects, the text generation unit 45 generates text information based on the tacit knowledge model 4004, the target image, and voice information and character information, which are examples of input information accepted by the input acceptance unit 12 when the display screen 1000 is displayed on the display 106a.

[0158] This allows the server 40 to generate text information that is more in line with the user's intentions, based on the input voice information, character information, and the like.

[0159] [Sixth aspect] In the fifth embodiment, when the voice information and the character information, which are examples of the input information, include a question, the text information includes an answer to the question. This allows the server 40 to generate text information that is more in line with the user's intention in response to the input question.

[0160] [Seventh aspect] In any of the first to sixth aspects, the device further includes an update unit 46 which is an example of a model update means for updating the tacit knowledge model 4004 using, as learning data, a specified target image in the display screen 1000 displayed on the display 106a and text data based on at least one of the voice information and the character information received by the input receiving unit 12.

[0161] As a result, the server 40 can generate highly accurate text information by updating the tacit knowledge model 4004 using the target image and text data as learning data.

[0162] [Eighth aspect] In the seventh aspect, the update unit 46 updates the tacit knowledge model 4004 based on at least one of user information that identifies a user, acquisition information that identifies an acquisition scene in which the image that is the basis of the display screen 1000 was acquired, and usage information that identifies a usage scene in which the display screen 1000 is used.

[0163] As a result, the server 40 can generate more accurate text information by including user attributes based on user information, acquired information, and usage information in the learning data.

[0164] [Ninth aspect] In the eighth aspect, the update unit 46 updates the tacit knowledge model 4004 based on the relevance between the target image and the text data.

[0165] As a result, the server 40 can generate more accurate text information by updating the tacit knowledge model 4004 based on the degree of association between the target image and the text data.

[0166] [Tenth aspect] In the ninth aspect, the update unit 46 updates the tacit knowledge model 4004 based on the caption comment explaining the content of the target image and the relevance of the text data.

[0167] As a result, the server 40 can generate more accurate text information by updating the tacit knowledge model 4004 based on the caption comment explaining the content of the target image and the relevance of the text data.

[0168] [Eleventh aspect] The terminal device 10, which is an example of an input / output device according to an embodiment of the present invention, includes an input receiving unit 12 that receives a voice, a character, or an operation that identifies a target image on a display screen 1000 displayed on a display 106a, and at least one of a display control unit 13 and a voice control unit 14 that are examples of output means that output text information generated by a text generation unit 45 based on an implicit knowledge model 4004 generated by executing a learning process using an image and text as learning data, and the target image.

[0169] [12th aspect] An information processing method according to one embodiment of the present invention is an information processing method executed by the server 40, which executes a text information generation step S11 of generating text information based on a tacit knowledge model 4004 generated by executing a learning process using a target image identified from the display screen 1000 displayed on the display 106a, and the image and text as learning data.

[0170] [13th aspect] A program according to one embodiment of the present invention causes a computer to execute a text information generation step S11 in which text information is generated based on a tacit knowledge model 4004 generated by executing a learning process using a specified target image on the display screen 1000 displayed on the display 106a, and the image and text as learning data.

[0171] [14th aspect] A method according to one embodiment of the present invention is a method for performing input and output by the terminal device 10, which includes an input receiving step S7 for receiving a voice, character, or operation for identifying a target image on the display screen 1000 displayed on the display 106a, and an output step S13 for outputting text information generated by the text generating unit 45 based on the target image and the tacit knowledge model 4004 generated by executing a learning process using the image and text as learning data.

[0172] [15th aspect] A program according to one embodiment of the present invention causes a computer to execute an input receiving step S7 for receiving a voice, character, or operation for identifying a target image on the display screen 1000 displayed on the display 106a, and an output step S13 for outputting text information generated by the text generating unit 45 based on the target image and the tacit knowledge model 4004 generated by executing a learning process using the image and text as learning data.

[0173] [16th aspect] A model according to one embodiment of the present invention is a tacit knowledge model 4004 that is generated by executing a learning process using images and text as learning data and causes a computer to function to output text based on images, and outputs text to be used by the text generation unit 45 to generate text information based on a target image identified on the display screen 1000 displayed on the display 106a.

[0174] [17th aspect] An information processing system 1 according to an embodiment of the present invention includes a terminal device 10 and a server 40 capable of communicating with the terminal device 10. The terminal device 10 includes an input receiving unit 12 that receives a voice, character, or operation that identifies a target image on a display screen 1000 displayed on the display 106a, and a transmission / reception unit 11 that transmits the target image to the server 40. The server 40 includes a transmission / reception unit 41 that receives the target image transmitted from the terminal device 10, and transmits to the terminal device 10 an implicit knowledge model 4004 generated by performing a learning process using the image and text as learning data, and text information generated based on the target image. The terminal device 10 further includes at least one of a display control unit 13 and a voice control unit 14 that receive text information from the server 40 via the transmission / reception unit 11 and output the text information. [Explanation of symbols]

[0175] 1. Information Processing Systems 100 Communication Network 10 Terminal equipment (an example of an input / output device) 11 Transmitting / receiving unit (an example of a transmitting means or a receiving means) 12 Input reception unit (an example of an input reception means) 13 Display control unit (an example of a display control means and an output means) 14 Audio control unit (an example of audio control means and output means) 15 Processing section 19 Memory / read unit (an example of a memory control means) 106a Display (an example of a display unit) 1000 Memory unit (an example of a memory means) 40 Server (an example of an information processing device) 41 Transmitting / receiving unit (an example of a transmitting means or a receiving means) 42 Screen generation unit (an example of a screen generation means) 43 Processing unit (an example of a three-dimensional information generating means) 43 Judgment unit (an example of a judgment means) 44 Specific unit (an example of specific means) 45 Text generation unit (an example of a text generation means) 46 Update unit (an example of an update means) 49 Memory / read unit (an example of a memory control means) 4000 Memory unit (an example of a memory means) 4001 User information management DB (an example of user information management means) 4002 Image management DB (an example of image management means) 4003 Caption Model 4004 Tacit Knowledge Model 4005 Large-scale language model 1000 display screen 1100, 1110 Images 1100A 1st image 1100B 2nd image 1100B1 Partial image 1200, 1210 Text

Claims

1. A storage means for storing a model generated by executing a learning process using images and text as learning data; a text information generating means for generating text information based on a specified target image among the display screens displayed on a display unit and the model; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the text information generating means generates the text information based on user information for identifying a user.

3. The information processing apparatus according to claim 1 , wherein the display screen is associated with at least one of acquisition information identifying an acquisition scene in which an image used to generate the display screen was acquired, and usage information identifying a usage scene in which the display screen is used.

4. 2. The information processing device according to claim 1, wherein the text information generation means generates the text information based on at least one of acquisition information that identifies an acquisition scene in which the image that is the basis of the display screen is acquired, and usage information that identifies a usage scene in which the display screen is used.

5. The text information generating means The model; The target image; 2. The information processing apparatus according to claim 1, wherein the text information is generated based on input information received by an input receiving means when the display screen is displayed on the display unit.

6. The information processing apparatus according to claim 5 , wherein when the input information includes a question, the text information includes an answer to the question.

7. 2. The information processing apparatus according to claim 1, further comprising a model updating means for updating the model using a specified target image among the display screen displayed on the display unit and text data based on at least one of voice information and character information received by the input receiving means as the learning data.

8. The information processing device according to claim 7, wherein the model update means updates the model based on at least one of user information identifying a user, acquisition information identifying an acquisition scene in which the image on which the display screen is based was acquired, and usage information identifying a usage scene in which the display screen is used.

9. The information processing apparatus according to claim 8 , wherein the model update means updates the model based on a degree of association between the target image and the text data.

10. 10. The information processing apparatus according to claim 9, wherein the model update means updates the model based on a degree of association between a caption comment explaining the content of the target image and the text data.

11. an input receiving means for receiving a voice, a character, or an operation for identifying a target image on a display screen displayed on the display unit; an output means for outputting text information generated by the text information generating means based on a model generated by executing a learning process using an image and text as learning data and the target image; An input / output device equipped with

12. An information processing method executed by an information processing device, An information processing method that executes a text information generation step of generating text information based on a model generated by executing a learning process using an identified target image from a display screen displayed on a display unit, and the image and text as learning data.

13. A program that causes a computer to execute a text information generation step of generating text information based on a model generated by executing a learning process using an identified target image among the display screens displayed on a display unit, and the image and text as learning data.

14. 1. A method for performing input / output by an input / output device, comprising: an input receiving step of receiving a voice, a character, or an operation for identifying a target image on a display screen displayed on the display unit; an output step of outputting text information generated by a text information generating means based on a model generated by executing a learning process using an image and text as learning data and the target image; How to do it.

15. an input receiving step of receiving a voice, a character, or an operation for identifying a target image on a display screen displayed on the display unit; an output step of outputting text information generated by a text information generating means based on a model generated by executing a learning process using an image and text as learning data and the target image; A program that causes a computer to execute the following.

16. A model that is generated by executing a learning process using an image and a tacit knowledge comment for the image as learning data, and causes a computer to function so as to output a tacit knowledge comment based on the image, A model that outputs a tacit knowledge comment to be used for generating text information by a text information generating means, based on a specified target image among the display screens displayed on a display unit.

17. An information processing system including an input / output device and an information processing device capable of communicating with the input / output device, The input / output device is an input receiving means for receiving a voice, a character, or an operation for identifying a target image on a display screen displayed on the display unit; a transmission means for transmitting the target image to the information processing device, The information processing device includes: a receiving means for receiving the target image transmitted from the input / output device; a transmission means for transmitting a model generated by executing a learning process using an image and a text as learning data, and text information generated based on the target image, to the input / output device; The input / output device is A receiving means for receiving the text information from the information processing device; an output means for outputting the text information; The information processing system further comprises:

Images