Information processing system, first server, second server, information processing method, program

Abstract

To perform processing based on information managed by the first server and information managed by the second server, without adding processing functions to the first server. [Solution] The second server includes a processing unit 47 that identifies a captured image based on the field of view of the three-dimensional image information selected by the terminal device, obtains text data acquired when the captured image was taken from the first server, and performs a process of associating the text data with the three-dimensional image information of the field of view, or associating generated information generated based on the text data with the three-dimensional image information of the field of view. The terminal device displays the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information, received from the second server on the display screen.

Description

Technical Field

[0001] The present invention relates to an information processing system, a first server, a second server, an information processing method, and a program.

Background Art

[0002] There are cases where a first server and a second server respectively manage related information. In such cases, a technique is known in which a terminal device displays information managed by the first server and information managed by the second server (see, for example, Patent Document 1).

[0003] Patent Document 1 discloses a system that displays related information of a property transmitted from a link information management system and a full-spherical image of the property transmitted from an image management system on a communication terminal.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention provides a technique for performing processing based on information managed by a first server and information managed by a second server without adding a processing function to the first server.

Means for Solving the Problems

[0005] The present invention is an information processing system including a first server that manages text data based on voice data acquired when a photographing device photographs a photographed image of an object, three-dimensional image information of the object, and a second server that manages the photographed image aligned with the three-dimensional image information, and a terminal device capable of communicating with the first server and the second server, The terminal device includes a display control unit that displays a display screen including the text data received from the first server and the three-dimensional image information received from the second server. The second server includes a processing unit that identifies the captured image based on the field of view of the three-dimensional image information selected by the terminal device, retrieves the text data acquired when the captured image was taken from the first server, and performs a process to associate the text data with the three-dimensional image information of the field of view, or associate generated information generated based on the text data with the three-dimensional image information of the field of view. The display control unit of the terminal device displays the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information, received from the second server, on the display screen. [Effects of the Invention]

[0006] Without adding processing functions to the first server, processing can be performed based on information managed by the first server and information managed by the second server. [Brief explanation of the drawing]

[0007] [Figure 1] This is an overall configuration diagram of an example of an information processing system. [Figure 2] This is a hardware configuration diagram of an example of an image management server, conference management server, or terminal device. [Figure 3] This is an example functional configuration diagram illustrating the functions of an image management server, a conference management server, and terminal devices in an information processing system. [Figure 4] This is a conceptual diagram showing an example of a three-dimensional image information management table according to this embodiment. [Figure 5] This is a conceptual diagram showing an example of a captured image information management table according to this embodiment. [Figure 6] This is a conceptual diagram showing an example of a meeting information management table according to this embodiment. [Figure 7] This is a sequence diagram illustrating an example of communication processing between wide-field images and audio data. [Figure 8] This is an explanatory diagram displayed by the terminal device during text information generation processing and model update processing. [Figure 9] This is an explanatory diagram displayed by the terminal device during text information generation processing and model update processing. [Figure 10] This sequence diagram shows an example of a process that generates screen information by arranging audio text and captured images or three-dimensional image information. [Figure 11] This is a diagram showing an example of a property selection screen. [Figure 12] This diagram shows an example of a voice-to-text display screen. [Figure 13] This diagram shows an example of a text image display screen. [Figure 14] This diagram shows an example of a screen displaying past text and past screenshots. [Figure 15] This diagram illustrates the display screen for past text and past captures when a user selects a different voice-to-text option. [Figure 16] This sequence diagram shows an example of a process that generates screen information by arranging audio text and captured images or three-dimensional image information. [Figure 17] This is a sequence diagram showing an example of a model update process. [Figure 18] This figure shows an example of a text image display screen shown by a terminal device. [Figure 19] This is an example sequence diagram illustrating the text information generation process. [Figure 20] This figure shows an example of a text-image display screen during the inference phase. [Figure 21] This figure shows an example of text information displayed on the past text and past capture screen. [Figure 22] This is an example functional configuration diagram illustrating the functions of an image management server, a conference management server, and terminal devices in an information processing system. [Figure 23] This is an example sequence diagram showing the process of generating text and image information. [Figure 24]It is a diagram showing the generated image information displayed on the past text past capture display screen.

Mode for Carrying Out the Invention

[0008] Hereinafter, as an example of a mode for carrying out the present invention, an information processing system and an information processing method performed by the information processing system will be described while referring to the drawings.

[0009] <Supplement Regarding Tacit Knowledge> In industries such as civil engineering and architecture, BIM / CIM implementation is being promoted for purposes such as addressing the declining birthrate and aging population, and improving labor productivity.

[0010] BIM is an abbreviation for Building Information Modeling. It is a solution for leveraging information in all processes from building design, construction to maintenance management, using a building database that adds attribute data such as cost, finish, and management information to a three-dimensional digital model of a building (hereinafter referred to as a 3D model) created on a computer.

[0011] CIM is an abbreviation for Construction Information Modeling. It is a solution for the civil engineering field (covering all infrastructure such as roads, electricity, gas, water supply, etc.) proposed following BIM, which has been promoted in the construction field. Similar to BIM, CIM aims to streamline and enhance a series of construction production systems by sharing information among stakeholders centered around a 3D model.

[0012] What is important in promoting BIM and CIM implementation is how to utilize the constructed BIM and CIM.

[0013] Specifically, the 3D model restored by BIM and CIM can be utilized not only for design and construction purposes, but also for other tasks such as maintenance management and site surveys. That is, not only can it be used as a design drawing, but other uses such as leaving records in the 3D model and sharing with others are conceivable.

[0014] Furthermore, since the work performed on the 3D model can be recorded as a log, if tacit knowledge can be extracted from this data, it can be effectively used for transferring skills from experienced personnel to younger ones. This is expected to lead to front-loading of operations and talent development.

[0015] Focusing on the transmission of tacit knowledge, the challenge lies not only in 3D models but also in 2D data (such as 3D spherical images and planar images), as mentioned above, in how to transmit tacit knowledge between different tasks and among users with varying levels of skill.

[0016] Specifically, because tacit knowledge is qualitative and difficult to quantify, even if an tacit knowledge model is generated from tacit knowledge, it is difficult to secure user trust in the tacit knowledge model, making it difficult to promote its use. For example, if the field of expertise of the tacit knowledge model differs from that of the user, no matter how excellent the tacit knowledge model is, it will have no value to the user. Similarly, if the knowledge level of the tacit knowledge model is lower than the user's knowledge level, it will have no value to the user.

[0017] However, it is also true that tacit knowledge models can give users new perspectives and insights, and by using tacit knowledge models, even inexperienced users have great potential to acquire know-how and skills and apply them to their work.

[0018] Furthermore, in a system that includes a first server and terminal device that stores audio text acquired during meetings about a property, it may be desirable to add a function to display at least one of either three-dimensional image information such as a 3D model corresponding to the property or captured images from the meeting.

[0019] One way to achieve this is for the first server to acquire at least one of the following: 3D image information such as 3D models corresponding to the property, and captured images during the meeting. However, this would require adding functionality to the first server, leading to increased costs.

[0020] Therefore, in this embodiment, the second server performs processing based on at least one of the audio text managed by the first server and the three-dimensional image information or captured image managed by the second server. This processing includes displaying at least one of the audio text managed by the first server and the three-dimensional image information or captured image managed by the second server on a single screen.

[0021] Furthermore, the second server can not only display two pieces of information on the terminal device, but can also display tacit knowledge about the object (such as text information) generated based on at least one of the captured image and the three-dimensional image information, in association with the three-dimensional image information or the captured image. This allows the terminal device to display at least one of the three-dimensional image information or the captured image and audio text in association on a single screen, or to display tacit knowledge about the object in association with at least one of the three-dimensional image information or the captured image, without adding significant functionality to the first server.

[0022] <About Terminology> A user is someone who uses the text information (or other content such as images) generated by the tacit knowledge model. A data provider is someone who provides the data (such as audio information, text information, operation information, images, 3D data, etc.) that the tacit knowledge model uses for training.

[0023] Tacit knowledge is knowledge based on an individual's experience, intuition, etc. An tacit knowledge model is a model that learns tacit knowledge and outputs answers based on that learned tacit knowledge in response to questions. A model is a mechanism or artificial intelligence (AI) that learns the correspondence between input data and output data and outputs output data for given input data. Note that the output data does not depend on whether or not there is training data.

[0024] A property can be any space where items can be placed, such as a facility or a room within a facility. Items are things that are placed in the property. The types of items placed will vary depending on the function of the facility.

[0025] For example, properties include real estate, factories, construction sites, research facilities, medical facilities, farmland, warehouses, and equipment requiring maintenance. Goods include furniture, materials, equipment, heavy machinery, tools, instruments, materials, cultured products, and food.

[0026] Three-dimensional image information of an object is an image of a 3D model captured by a virtual camera. The user can change the viewpoint of the three-dimensional image information.

[0027] The generated information is information generated based on three-dimensional image information and captured images. The generated information may be generated by an implicit knowledge model. In this embodiment, it is described using the terms implicit knowledge comments or text information.

[0028] A display screen is, for example, a screen in which audio text and captured images or three-dimensional image information, or generated information and captured images or three-dimensional image information are displayed on a single screen. In this embodiment, the screens shown in Figures 14 and 21 are examples.

[0029] A wide-field image is an image with a wide field of view that includes areas that cannot be seen with a normal field of view. For example, a wide-field image is an image with a wide field of view that is captured with a wide shooting range, including a 360-degree image (also called a spherical image, omnidirectional image, or all-around image) that captures the entire 360 ​​degrees.

[0030] A predetermined region image is an image corresponding to a predetermined region within a wide-field image. The predetermined region image is projected onto a two-dimensional plane and is therefore a planar image. In this embodiment, the predetermined region image is saved by a capture operation and is therefore called a captured image.

[0031] [First Embodiment] <Example System Configuration> Figure 1 is an overall configuration diagram of an example of an information processing system 100. The information processing system 100 in this embodiment includes a terminal device 10, a shooting device 5, an image management server 40, and a conference management server 20, which are examples of input / output devices. However, the terminal device 10 may be connected to the image management server 40 or the conference management server 20 as needed, and is not required to be included in the information processing system 100.

[0032] The image management server 40 (an example of a second server) is one or more information processing devices that can communicate with the terminal device 10 via a communication network N. The image management server 40 manages three-dimensional image information and captured images of the object, and also has an implicit knowledge model and a large-scale language model. The image management server 40 uses these to return text information containing implicit knowledge to the user. The image management server 40 may be a web server that returns processing results to the terminal device 10 in response to requests from the terminal device 10. A server is a computer or software that performs the function of providing information and processing results in response to requests from a client.

[0033] The image management server 40 may support cloud computing. Cloud computing refers to a usage model in which network resources are utilized without being aware of specific hardware resources. Cloud computing can take the form of SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service), and any of these forms is acceptable. Therefore, the image management server 40 does not need to be housed in a single chassis or provided as a single unit of equipment. The image management server 40 may have its functions distributed among multiple information processing units, or multiple information processing units may each have all the functions, and the information processing unit being used may be switched through load balancing or other means. Furthermore, the image management server 40 may be a server located in an on-premises environment.

[0034] Furthermore, instead of the image management server 40 having both an implicit knowledge model and a large-scale language model, the image management server 40 may call an API (Application Programming Interface) exposed by an external system and utilize at least one of the implicit knowledge model or the large-scale language model.

[0035] The conference management server 20 (an example of the first server) is one or more information processing devices that can communicate with the terminal device 10 via the communication network N. The conference management server 20 manages the audio text spoken in meetings concerning the property. The conference management server 20 does not possess image information, or if it does, it is merely a photograph or similar image that is different from the images managed by the image management server 40.

[0036] The conference management server 20 may be a web server that returns processing results to the terminal device 10 in response to requests from the terminal device 10. The conference management server 20 can communicate with the image management server 40 via the communication network N. The conference management server 20 may also support either cloud computing or on-premises deployment.

[0037] It is preferable that the image management server 40 and the conference management server 20 are coordinated to the extent that single sign-on is possible. The image management server 40 can communicate with the conference management server 20 via an API exposed by the conference management server 20. Alternatively, the image management server 40 and the conference management server 20 may have a business partnership.

[0038] Terminal device 10 is a general-purpose information processing terminal used by users of the information processing system 100. Terminal device 10 runs a web browser and native applications dedicated to the image management server 40 or conference management server 20. When terminal device 10 runs a web browser, terminal device 10 and the image management server 40 or conference management server 20 run a web application. A web application is an application that operates through the cooperation of a program written in a programming language (e.g., JavaScript®) that runs on a web browser and a program on the web server (image management server 40). When a web application is executed, the processing in this embodiment may be performed by the image management server 40 or conference management server 20, or by terminal device 10 that receives the web application.

[0039] An application that cannot be executed unless it is installed on the terminal device 10 is called a native application. In this embodiment as well, the application executed on the terminal device 10 may be either a web application or a native application. In this case as well, the processing in this embodiment may be performed by the image management server 40 or by the terminal device 10 that is running the native application.

[0040] Terminal device 10 can be, for example, a PC (Personal Computer), a smartphone, a PDA (Personal Digital Assistant), or a tablet device. Alternatively, terminal device 10 can be any device capable of running a web browser or native applications. Terminal device 10 may also be an electronic whiteboard, a television receiver, a glasses device, or a wearable device. Furthermore, there may be multiple terminal devices 10.

[0041] Terminal device 10 can communicate with image management server 40 and conference management server 20 via communication network N. Communication network N is constructed using the Internet, LAN (Local Area Network), provider services, etc. Communication network N may include not only wired communication but also mobile communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), and wireless LAN networks. Furthermore, terminal device 10 can communicate using short-range communication technologies such as Bluetooth (registered trademark) and NFC (registered trademark, Near Field Communication).

[0042] The imaging device 5 is a digital camera for obtaining wide-field images and recording sound. The imaging device 5 is connected to the communication network N via the relay device 3. The relay device 3 has the function of a cradle for charging the imaging device 5 and transmitting and receiving data. In addition, the relay device 3 can communicate data with the imaging device 5 via contacts and can also communicate data with the conference management server 20 via the communication network N. The imaging device 5 and the relay device 3 are installed at predetermined locations on site Sa, such as construction sites, exhibition halls, educational institutions, and medical facilities. The imaging device 5 may be a general digital camera that obtains narrow-field images, such as a single-lens reflex camera, and the conference management server 20 may distribute live images of the narrow-field images captured by the imaging device 5. In this case, the predetermined area image is an image corresponding to all or part of a predetermined area of ​​the captured image.

[0043] In Figure 1, the image management server 40, the conference management server 20, and the terminal device 10 communicate via the communication network N. However, users may also operate the image management server 40 or the conference management server 20 directly from a console.

[0044] <Example Hardware Configuration> <<Image management server, conference management server, terminal devices>> Figure 2 is a hardware configuration diagram of an example of the image management server 40, conference management server 20, or terminal device 10 according to this embodiment. The hardware components of the image management server 40 or conference management server 20 are indicated by reference numerals in the 400 series. The hardware components of the terminal device 10 are indicated by reference numerals in the 100 series.

[0045] The hardware configuration of the terminal device 10 will be described below, but the hardware configuration of the image management server 40 or the conference management server 20 is the same and will therefore not be described.

[0046] The terminal device 10 is built using a computer and, as shown in Figure 2, includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HD (Hard Disk) 104, HDD (Hard Disk Drive) controller 105, display I / F 106, and communication I / F 107.

[0047] Of these components, the CPU 101 controls the overall operation of the terminal device 10. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101.

[0048] HD104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to HD10 according to the control of the CPU 101.

[0049] The display I / F 106 is a circuit that displays images on the display 106a. The display 106a is a type of display unit such as a liquid crystal or organic EL (electroluminescence) that displays various information such as cursors, menus, windows, characters, or images. The communication I / F 107 is an interface used for communication with other devices.

[0050] If the terminal device 10 is a glass device, the terminal device 10 may use a circuit that displays an image on a lens or the like, which is a transmissive reflective material, instead of the display I / F 106.

[0051] Communication I / F107 is, for example, a NIC (Network Interface Card) that supports TCP (Transmission Control Protocol) / IP (Internet Protocol).

[0052] Furthermore, the terminal device 10 is equipped with a sensor I / F 108, an audio input / output I / F 109, an input I / F 110, a media I / F 111, and a DVD-RW (Digital Versatile Disk Rewritable) drive 112.

[0053] The sensor interface 108 is an interface for receiving detection information from various sensors. The sound input / output interface 109 is a circuit that processes the input and output of sound signals between the speaker 109a and the microphone 109b according to the control of the CPU 101. The input interface 110 is an interface for connecting a predetermined input means to the terminal device 10.

[0054] Keyboard 110a is a type of input device equipped with multiple keys for inputting characters, numbers, various instructions, etc. Mouse 110b is a type of input device used for selecting and executing various instructions, selecting processing targets, moving the cursor, and operating on the display screen, etc.

[0055] The media interface 111 controls the reading or writing (storage) of data to or from a recording medium 111a such as flash memory. The DVD-RW drive 112 controls the reading or writing of various types of data to or from a DVD-RW 112a, which is an example of a removable recording medium. Note that the DVD-RW may be a DVD-R or the like. The DVD-RW drive 112 may also be a Blu-ray drive that controls the reading or writing of various types of data to or from a Blu-ray Disc (registered trademark).

[0056] Furthermore, the terminal device 10 is equipped with a bus line 113. The bus line 113 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 101.

[0057] Furthermore, recording media such as HDs and CD-ROMs on which the above programs are stored can be provided domestically or internationally as program products. The terminal device 10 realizes the information processing method according to the present invention by executing, for example, the program according to the present invention.

[0058] <About the features> Figure 3 is a functional configuration diagram illustrating an example of the functions of the image management server 40, the conference management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment. The imaging device 5 and the relay device are assumed to have existing functions.

[0059] <<Terminal device>> As shown in Figure 3, the terminal device 10 includes a transmitting / receiving unit 11, an input receiving unit 12, a display control unit 13, an audio control unit 14, a conversion unit 15, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 101 that follow a program deployed from the HD 104 onto the RAM 103. The terminal device 10 also has a storage unit 1000 constructed from at least one of the RAM 103 and HD 104 shown in Figure 2.

[0060] The transmitting / receiving unit 11 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 101 shown in Figure 2, as well as the communication interface 107, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0061] The input receiving unit 12 is an example of an input receiving means and is mainly implemented by commands from the CPU 101 shown in Figure 2, as well as the input I / F 110 and the sound input / output I / F 109, and accepts various inputs from the user using the microphone 109b, keyboard 110a and mouse 110b.

[0062] The display control unit 13 is an example of a display control means and output means, and is implemented by commands from the CPU 101 shown in Figure 2 and the display I / F 106, and causes various images and screens to be displayed on the display 106a, which is an example of a display unit. If the terminal device 10 is a glasses device, the display control unit 13 causes a virtual image to be displayed on a transmissive reflective member such as a lens instead of the display I / F 106.

[0063] The audio control unit 14 is an example of an audio control means and output means, and is implemented by commands from the CPU 101 shown in Figure 2 and the audio input / output I / F 109, causing the speaker 109a, which is an example of an audio playback unit, to play sound.

[0064] The conversion unit 15 is an example of a processing means, and is implemented by instructions from the CPU 101 shown in Figure 2. It performs processing to convert text information into audio information and processing to convert audio information into text information.

[0065] The storage / reading unit 19 is an example of a storage control means and is executed by instructions from the CPU 101 shown in Figure 2, as well as by the HD 104, media I / F 111, and DVD-RW drive 112. It performs processing such as storing various data in the storage unit 1000, recording media 111a, and DVD-RW 112a, and reading various data from the storage unit 1000, recording media 111a, and DVD-RW 112a.

[0066] <Image management server functional configuration> The image management server 40 includes a transmission / reception unit 41, a screen generation unit 42, a determination unit 43, a identification unit 44, a text information generation unit 45, an update unit 46, a processing unit 47, and a storage / reading unit 49. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 following a program deployed from the HD 404 onto the RAM 403. The image management server 40 also has a storage unit 4000 constructed from the HD 404 shown in Figure 2. The storage unit 4000 is an example of a storage means.

[0067] In Figure 3, all functions are installed in a single image management server 40, but the image management server 40 may also be configured to distribute each function across multiple computers.

[0068] The transmitting / receiving unit 41 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0069] The screen generation unit 42 is an example of a screen generation means and is implemented by instructions from the CPU 401 shown in Figure 2, generating various screens. When the terminal device 10 runs a web application, screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc. Therefore, screen information may also be referred to as a web application. When the terminal device 10 runs a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0070] The decision unit 43 is an example of a decision-making mechanism, and is implemented by instructions from the CPU 401 shown in Figure 2, and performs various decisions described later.

[0071] The identification unit 44 is an example of an identification means, and is implemented by instructions from the CPU 401 shown in Figure 2, and identifies the target image.

[0072] The text information generation unit 45 is an example of a text information generation means, and is implemented by instructions from the CPU 401 shown in Figure 2. It obtains implicit knowledge comments from an implicit knowledge model and generates text information based on a large-scale language model 4005.

[0073] The update unit 46 is an example of an update means, and is implemented by instructions from the CPU 401 shown in Figure 2, and performs an update of the implicit knowledge model, which will be described later.

[0074] The processing unit 47 performs processing to associate three-dimensional image information or captured images with audio text, or three-dimensional image information or captured images with generated information (an example of text information) generated from the three-dimensional image information and captured images, in accordance with the processing requested by the user. The matching process includes processing to display the three-dimensional image information or captured images with audio text, or three-dimensional image information or captured images with generated information on a single screen. The matching process also includes processing to obtain generated information, which is an example of text information, from the tacit knowledge model 4004 using the three-dimensional image information and captured images, etc. The processing unit 47 requests processing from the screen generation unit 42 or the text information generation unit 45, etc., depending on the content of the processing.

[0075] The storage / reading unit 49 is an example of a storage control means and is executed by instructions from the CPU 401 shown in Figure 2, as well as by the HD 404, media I / F 411, and DVD-RW drive 412. It performs processing such as storing various data in the storage unit 4000, recording media 411a, and DVD-RW 412a, and reading various data from the storage unit 4000, recording media 411a, and DVD-RW 412a. The storage unit 4000, recording media 411a, and DVD-RW 412a are examples of storage means.

[0076] The memory unit 4000 contains a three-dimensional image information management DB 4001, a model shape management DB 4002, a caption model 4003, an implicit knowledge model 4004, a large-scale language model 4005, and a captured image information management DB 4006.

[0077] The 3D image information management DB 4001 manages 3D image information of items placed on the property. This 3D image information is information that visually represents the items (also called models) placed on the property. The model shape management DB 4002 manages 3D model shape information of items placed on the property. The image management server 40 can generate image information related to the property using the 3D model shape information. The 3D model shape information is information that renders the item in 3D, such as a 3D model or 3D point cloud. The 3D model shape information may be composed of polygons or CAD. In addition, the 3D image information management DB 4001 or the model shape management DB 4002 may store wide-field images such as 360-degree images of the property.

[0078] Caption model 4003 is a model that is generated by performing a training process using image and caption comment combinations as training data, and enables a computer to function to output caption comments based on images. Caption comments are explicit knowledge and are used as a term in contrast to tacit knowledge. Here, caption comments are text data, such as comments expressed in audio or text that describe an image. Caption comments relating to objects or articles are associated with the identification information of those objects or articles.

[0079] The tacit knowledge model 4004 is generated by performing a learning process using the correspondence between three-dimensional image information and captured images and the tacit knowledge (input information, speech text, etc.) associated with this three-dimensional image information and captured images as training data. It is a model that enables a computer to function to output tacit knowledge comments based on images. The tacit knowledge model 4004 learns by associating information as follows. Correspondence between three-dimensional image information and captured images and input information • Correspondence between three-dimensional image information and captured images and audio text. Correspondence between three-dimensional image information and captured images, and audio text and input information. Here, tacit knowledge comments are text data, and are comments expressed in audio or text, excluding caption comments, that is, comments relating to content not represented in the image.

[0080] The large-scale language model 4005 is a computer language model that is generated by performing a training process using a vast amount of unlabeled text as training data and consists of an artificial neural network with many parameters. The large-scale language model 4005 can capture much of the syntax and meaning of human language by being sufficiently trained with context-learning techniques such as next sentence prediction, which understands context by determining whether sentence 1 and sentence 2 are consecutive, and the masked language model, which understands context by masking words in a sentence and predicting the masked words from the words before and after them.

[0081] The captured image information management DB4006 manages wide-field images captured by the camera 5 during meetings related to the property, etc., in chronological order. These wide-field images may be videos. In addition, if a user of the communication terminal, described later, performs a capture operation, the captured image is saved. Capture refers to saving a predetermined area of ​​the wide-field image, which represents a predetermined area, as a still image. Furthermore, the captured image information management DB4006 manages audio text acquired from the meeting management server. This audio text is text data converted from audio data recorded by the camera 5 or the communication terminal during meetings, etc.

[0082] <<Three-dimensional image information management table>> Figure 4 is a conceptual diagram showing an example of a three-dimensional image information management table according to this embodiment. The storage unit 4000 has a three-dimensional image information management DB 4001 constructed, which is composed of the three-dimensional image information management table shown in Figure 4. In the three-dimensional image information management table shown in Figure 4, the model ID and location information are associated and managed in relation to the property identification information.

[0083] Property identification information is an example of property identification information used to identify a property. A property can be any space where goods can be placed, such as a facility or a room within a facility. The types of goods that can be placed will vary depending on the function of the facility. A property can be represented in an easily manageable unit, such as "○○ Building 2F-N (North side of the 2nd floor)".

[0084] A Model ID is an example of model identification information used to identify items placed in a property. The items may consist of three-dimensional model shape information such as polygons or CAD data in the Model Shape Management DB4002. The Model ID links the three-dimensional image information to the three-dimensional model shape in the Model Shape Management DB4002.

[0085] Location information is information that indicates the position of an item's model in a three-dimensional virtual space, where the item is represented in a virtual space, using three-dimensional coordinates (XYZ). For example, location information is indicated by the three-dimensional coordinates of eight points that define the rectangular space occupied by the model.

[0086] This location information is determined as the location information (latitude, longitude, and altitude) of the relay device 3 by GNSS (Global Navigation Satellite System) satellites such as GPS (Global Positioning Systems), or by IMES (Indoor Messaging System) as an indoor GPS. Indoor positioning methods include Wi-Fi positioning, RFID (Radio Frequency Identifier) ​​positioning, beacon positioning, pedestrian autonomous navigation positioning, geomagnetic positioning, acoustic positioning, and UWB (Ultra Wide Band) positioning.

[0087] As explained above, the positional information in Figure 4 is managed in correspondence with absolute positions on Earth. For example, by associating the origin of the positional information in Figure 4 (X=0, Y=0, Z=0) with absolute positions on Earth (latitude, longitude, altitude), all coordinates in the 3D image, including the 3D model and parts, are associated with absolute positions on Earth. In other words, the 3D image and the captured image are aligned.

[0088] <<Image Information Management Table>> Figure 5 is a conceptual diagram showing an example of a captured image information management table according to this embodiment. The memory unit 4000 has a captured image information management DB 4006 constructed, which is composed of the captured image information management table shown in Figure 5. In the captured image information management table shown in Figure 5, the date and time of shooting, wide-field image, captured image, shooting location, field of view information, and voice text of the current date and time are associated with the object identification information, stored, and managed. The position of the shooting device 5 is determined by the GNSS or the like of the relay device 3 to which the shooting device 5 is attached. The "Shooting Date and Time" indicates the date and time when the captured image was taken by the shooting device 5. One or more captured images are stored associated with the shooting date and time. The shooting position indicates the position of the shooting device 5 (absolute position on Earth) at the time the captured image was captured. The captured image is directly stored by the image management server 40. • Field of view information is information used to identify a predetermined region of a wide-field image that is displayed on a communication terminal (a terminal used to view wide-field images in real time during a conference, as described later). The audio text registered in the "Current Date and Time Audio Text" field is generated by speech recognition of the audio collected by the recording device 5. The current date and time audio text is the audio text sent from the conference management server 20.

[0089] <Functional Configuration of the Conference Management Server> Let's return to Figure 3 for explanation. The conference management server 20 has a transmitting / receiving unit 21, a screen generation unit 22, and a storage / reading unit 29. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 following a program deployed from the HD 404 onto the RAM 403. The conference management server 20 also has a storage unit 2000 constructed from the HD 404 shown in Figure 2. The storage unit 2000 is an example of a storage means.

[0090] In Figure 3, all functions are installed on a single conference management server 20, but the conference management server 20 may also be configured to distribute each function across multiple computers.

[0091] The transmitting / receiving unit 21 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0092] The screen generation unit 22 is an example of a screen generation means and is implemented by instructions from the CPU 401 shown in Figure 2, generating various screens. When the terminal device 10 runs a web application, screen information is created using HTML, XML, CSS, and JavaScript (registered trademark), etc. Therefore, screen information may also be referred to as a web application. When the terminal device 10 runs a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0093] The storage / reading unit 29 is an example of a storage control means and is executed by instructions from the CPU 401 shown in Figure 2, as well as by the HD 404, media I / F 411, and DVD-RW drive 412. It performs processing such as storing various data in the storage unit 2000, recording media 411a, and DVD-RW 412a, and reading various data from the storage unit 2000, recording media 411a, and DVD-RW 412a. The storage unit 2000, recording media 411a, and DVD-RW 412a are examples of storage means.

[0094] <<Meeting Information Management Table>> Figure 6 is a conceptual diagram showing an example of a meeting information management table according to this embodiment. The storage unit 2000 has a meeting information management DB 2001 constructed, which is composed of meeting information management tables as shown in Figure 6.

[0095] In this meeting information management table, the date and time of sound collection, the current date and time of audio text (recording device), and the current date and time of audio text (communication terminal) are associated with and stored and managed in relation to the property identification information. The sound collection date and time indicates the date and time when the audio was collected by the recording device 5 or the communication terminal 9. The audio text registered in the "Current Time Audio Text (Recording Device)" field is generated based on the audio collected by recording device 5. The audio text is comment data about items spoken by meeting participants while viewing the live image. The audio text registered in the "Current Time Audio Text (Communication Terminal)" field is generated based on the audio spoken by users viewing live images on their communication terminals. The audio text is comment data about items spoken by meeting participants while viewing live images.

[0096] <Transmission of wide-field images and audio data> Figure 7 is a sequence diagram showing the communication processing of wide-field images and audio data. In this embodiment, we will describe the case where the imaging device 5, participant A's communication terminal 9a, and participant B's communication terminal 9b are participating in the same remote communication. Steps S1 to S4 in Figure 7 are repeated.

[0097] S1: The camera 5 captures images of the surroundings and collects sound, sending video data (wide-field image) and audio data to the relay device 3. The camera 5 also transmits a device ID to identify the camera 5 in order to identify the property. As a result, the relay device 3 acquires the video data and audio data. The image management server 40 has pre-associated device IDs with properties.

[0098] S2: The relay device 3 transmits the acquired video data, audio data, and device ID to the image management server 40 via the communication network N. The image management server 40 then receives the video data, audio data, and device ID via the transmitting / receiving unit 41. The image management server 40 identifies the object using the device ID. As a result, the wide-field image and the date and time of capture are stored, for example, every second, in the captured image information management DB 4006 by the storage / reading unit 49. Note that live images may not be stored, and only distribution may occur.

[0099] S3a: The image management server 40 reads the participant IDs of participants in the same conference as the camera 5 from the conference information. The image management server 40 also reads the IP addresses of communication terminals 9a and 9b based on the read participant IDs. The image management server 40 refers to the IP address of communication terminal 9a and sends the received video data and audio data to communication terminal 9a. As a result, communication terminal 9a receives the video data and audio data, displays the wide-field image, and outputs sound.

[0100] S3b: Similarly, the image management server 40 refers to the IP address of the communication terminal 9b and sends video data and audio data to the communication terminal 9b. As a result, the communication terminal 9b displays the wide-field image and outputs sound.

[0101] S3c: The image management server 40 also sends audio data to the conference management server 20 by calling its API. The conference management server 20's transmitting / receiving unit 21 receives the audio data. The conference management server 20 (or an existing speech recognition server) uses the audio data to convert the audio portion into text data (hereinafter referred to as "audio text"). The storage / reading unit 29 stores the current date and time audio text (from the recording device) in the conference information management DB 2001.

[0102] S4a, 4b: Meanwhile, communication terminals 9a and 9b transmit audio data from participants A and B to the conference management server 20. This audio data is generated when microphones capture the voices emitted by participants A and B, who are operating communication terminals 9a and 9b, and convert them into audio data.

[0103] S4c: The image management server 40 sends audio data to the conference management server 20 by calling its API. The transmitting / receiving unit 21 of the conference management server 20 receives the audio data. The conference management server 20 (or an existing speech recognition server) uses the audio data to convert the audio portion into text and generate text data. The storage / reading unit 29 saves the current date and time audio text (communication terminal) to the conference information management DB 2001.

[0104] S5: Participants A and B (Participant B in the diagram) of communication terminals 9a and 9b can change the viewpoint of the video data, which is a wide-field image. If participant B wants to save a predetermined area of ​​the wide-field image displayed after changing the viewpoint, participant B can perform a capture operation at any time. Upon receiving the capture operation, communication terminal 9b sends a capture request and field of view information indicating the predetermined area currently displayed on the display to the image management server 40.

[0105] S6: When the image management server 40 receives the capture request and field of view information, it identifies the IP address of the relay device 3 which is participating in the same conference as the communication terminal 9b, and sends the capture request and field of view information to it.

[0106] S7: Relay device 3 receives the capture request and field of view information and transmits it to the shooting device 5.

[0107] S8: When the imaging device 5 receives a capture request, it generates a captured image based on the field of view information. The imaging device 5 transmits the captured image, shooting position, and field of view information to the relay device 3. If the imaging device 5 is fixed, the shooting position may be registered in advance with the image management server 40.

[0108] S9: The relay device 3 transmits the captured image, shooting location, and field of view information to the image management server 40. The image management server 40 identifies the property by the device ID, similar to step S3. The storage / reading unit 49 stores the captured image, shooting location, and field of view information in the captured image information management DB 4006.

[0109] Through the above processing, the captured image information management DB 4006 stores the captured image, the shooting position, and the field of view image captured from the wide-field image. The meeting information management DB 2001 stores the voice text transmitted by the shooting device 5 and the communication terminal 9. As will be described later, the voice text in the meeting information management DB 2001 may be sent to the captured image information management DB 4006.

[0110] <An example of model updating and text information generation> Refer to Figures 8 and 9 to explain how to update the model and generate text information. In Figures 8 and 9, check information is not used for updating the model and generating text information, but similar learning can be achieved by replacing or adding check information to dialogue Q1, etc. below.

[0111] Figure 8 is an explanatory diagram of what the terminal device 10 displays during text information generation processing and model update processing. Figure 8(a) is an explanatory diagram of the model update processing. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes the target image 1100 and text 1200.

[0112] The input receiving unit 12 of the terminal device 10 receives audio information from the microphone 109b, which represents the dialogue Q1, A1, Q2, and A2 between data providers M1 and M2, as input information to be entered by the data provider in response to the displayed screen 900. It is preferable that data providers M1 and M2 have extensive knowledge, including tacit knowledge, about their work. Through such a dialogue between data providers M1 and M2, the tacit knowledge model 4004 is updated, allowing the user to obtain useful tacit knowledge comments.

[0113] The identification unit 44 identifies the target image 1100, which is the portion of the display screen 900 excluding the text 1200.

[0114] The judgment unit 43 then uses the target image 1100 to determine the degree of relevance between the caption comments obtained from the caption model 4003 and the dialogues Q1, A1, Q2, and A2.

[0115] The update unit 46 updates the tacit knowledge model 4004 by using comments from the dialogues Q1, A1, Q2, and A2 that are judged to have a low degree of relevance as tacit knowledge comments, along with the target image 1100, as training data, and updates the caption model 4003 by using comments that are judged to have a high degree of relevance as caption comments, along with the target image 1100, as training data.

[0116] Therefore, the tacit knowledge model 4004 learns the correspondence between the target image 1100 and the dialogues Q1, A1, Q2, and A2. The target image 1100 has features extracted using several feature extraction models suitable for images, such as a CNN (Convolutional Neural Network). These features represent, for example, what is depicted and where, or the tasks being performed within the image. Thus, the tacit knowledge model 4004 can learn the correspondence between the image features and the dialogues Q1, A1, Q2, and A2.

[0117] Figure 8(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes an image 1110 and text 1210.

[0118] The input receiving unit 12 of the terminal device 10 receives audio information indicating questions Q11 and Q12 from user M3 via the microphone 109b as input information to be entered by the user on the displayed screen 900.

[0119] The identification unit 44 identifies the image 1110 that does not contain the text 1210 as the target image.

[0120] The text information generation unit 45 uses image 1110 to acquire tacit knowledge comments based on the tacit knowledge model 4004. The tacit knowledge model 4004 extracts features from image 1110 and determines that the features of image 1110 in Figure 8(b) are similar to those of image 1110 at the time of update, and can identify dialogues Q1, A1, Q2, and A2 related to image 1110. Dialogues Q1, A1, Q2, and A2 become tacit knowledge comments.

[0121] Furthermore, the text information generation unit 45 uses tacit knowledge comments (dialogue Q1, A1, Q2, A2) and questions Q11, Q12, etc., to generate text information related to the answers A11 and A12 to questions Q11 and Q12, respectively, based on the large-scale language model 4005.

[0122] The display control unit 13 of the terminal device 10 displays the text information related to the answers A11 and A12 received from the image management server 40 on the display 106a.

[0123] Figure 9 is another explanatory diagram displayed by the terminal device 10 during the model update process and text information generation process according to this embodiment. Figure 9 illustrates a case in which no question text is used during model update and text information generation.

[0124] Figure 9(a) is an explanatory diagram of the model update process. Figure 9(a) shows an example in which the tacit knowledge model 4004 is updated with audio information and partial images from a single data provider, rather than with conversations between data providers.

[0125] The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes a first image 1100A and a second image 1100B.

[0126] The input receiving unit 12 of the terminal device 10 receives character information from the keyboard 110a, which represents comments C1 to C4 from data provider M4, as input information to be entered by the data provider in response to the displayed screen 900.

[0127] Furthermore, the input receiving unit 12 receives operation information from the mouse 110b, which indicates an operation by the data provider M4 to identify a partial image 1100B1 in the second image 1100B, as input information to be entered by the data provider M4 into the displayed screen 900.

[0128] The identification unit 44 may identify the partial image 1100B1 as the target image, or it may identify the first image 1100A or the second image 1100B as the target image.

[0129] The judgment unit 43 then uses the target image to determine the degree of relevance between the caption comments obtained from the caption model 4003 and comments C1 to C4.

[0130] The update unit 46 updates the tacit knowledge model 4004 by using comments C1 to C4 that are judged to have low relevance as tacit knowledge comments, along with partial image 1100B1, etc., as training data, and updates the caption model 4003 by using comments that are judged to have high relevance as caption comments, along with partial image 1100B1, as training data.

[0131] Therefore, the tacit knowledge model 4004 learns the correspondence between the partial image 1100B1 and the comments C1-C4. Features are extracted from the partial image 1100B1 using several feature extraction models suitable for images, such as CNNs. These features represent, for example, what is depicted and where, or the content of the work being performed. Thus, the tacit knowledge model 4004 can learn the correspondence between the features of the image and the comments C1-C4.

[0132] Figure 9(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes the image 1110.

[0133] User M5 does not input anything to the displayed screen 900, and the input reception unit 12 does not accept any input information from the user to the displayed screen 900. The identification unit 44 identifies the entire displayed screen 900, which is image 1110, as the target image.

[0134] When user M5 performs an operation to identify a partial image 1100B1 on the display screen 900, the input receiving unit 12 receives operation information from the mouse 110b indicating the operation to identify the partial image as input information. In this case, the identification unit 44 identifies the partial image on the display screen 900 as the target image according to the operation information.

[0135] The text information generation unit 45 uses the partial image 1100B1 to acquire tacit knowledge comments based on the tacit knowledge model 4004. The tacit knowledge model 4004 determines that the features of image 1110B1 in Figure 9(b) are similar to those of image 1110B1 at the time of update, and can identify comments C1 to C4 related to image 1110B1. The tacit knowledge model 4004 extracts comments C1 to C4 as tacit knowledge comments.

[0136] The text information generation unit 45 generates text information related to comments C11 to C14 based on the large-scale language model 4005, using tacit knowledge comments and the like. The text information generation unit 45 may generate text information using preset standard questions if no question text is entered, rather than using a method that does not use any questions at all.

[0137] The display control unit 13 of the terminal device 10 displays the text information related to comments C11 to C14 received from the image management server 40 on the display 106a.

[0138] <Action or process> Next, we will describe a method for displaying audio-text and captured or three-dimensional image information on a single screen, as an example of processing based on these. In other words, the tacit knowledge model 4004 is not used.

[0139] Figure 10 is a sequence diagram showing an example of a process that generates screen information by arranging audio text and captured images or three-dimensional image information.

[0140] S11: The user enters a login operation into terminal device 10. This login is for the conference management server 20. The input reception unit 12 of terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful.

[0141] Users log in to the conference management server 20 first, and then log in to the image management server 40, but they may also log in to the image management server 40 first and then log in to the conference management server 20.

[0142] S12: Upon successful login, the transmitting / receiving unit 11 of the terminal device 10 sends a request for the property selection screen 200 to the conference management server 20.

[0143] S13: The transmitting / receiving unit 21 of the conference management server 20 receives a request for the property selection screen 200. The screen generation unit 22 generates the property selection screen 200, and the transmitting / receiving unit 21 transmits the screen information of the property selection screen 200 to the terminal device 10.

[0144] S14: The transmitting / receiving unit 11 of the terminal device 10 receives screen information from the property selection screen 200. The display control unit 13 displays the property selection screen 200 (see Figure 11). The user enters property identification information (for example, V0001, XX Building 2F-N) on the displayed property selection screen 200. The input receiving unit 12 of the terminal device 10 receives the property identification information.

[0145] S15: The transmitting / receiving unit 11 of the terminal device 10 sends a voice-to-text request to the conference management server 20, specifying the property identification information.

[0146] S16: The transmitting / receiving unit 21 of the conference management server 20 receives a voice-to-text request, and the storage / reading unit 29 searches the conference information management DB 2001 using the property identification information. The screen generation unit 22 of the conference management server 20 generates a voice-to-text display screen 210 that displays the voice-to-text, and the transmitting / receiving unit 21 transmits the screen information of the voice-to-text display screen 210 to the terminal device 10.

[0147] Furthermore, the transmitting / receiving unit 21 sends an image request program to the terminal device 10 in response to a voice-text request, enabling the terminal device 10 to acquire three-dimensional image information. The image request program is, for example, a web application. This web application is installed on the conference management server 20 by the operator of the image management server 40 with the permission of the operator of the conference management server 20. Alternatively, a URL containing the image request program may be sent to the terminal device 10. Since the web application acquires three-dimensional image information from the image management server 40, the terminal device 10 has the function of connecting to the image management server 40 and requesting or displaying three-dimensional image information.

[0148] S17: The transmitting / receiving unit 11 of the terminal device 10 receives screen information and an image request program for the voice-text display screen 210. The display control unit 13 displays the voice-text display screen 210 (see Figure 12). This displays the voice text related to the property. The user selects an arbitrary voice text from the displayed voice-text display screen 210. The user can select a voice text based on the name of the item contained in the voice text. The purpose of selecting a voice text is to display the three-dimensional image information and captured image identified by the voice text. When a voice text is selected, the date and time of recording are also identified. The input receiving unit 12 of the terminal device 10 accepts the operation to select a voice text.

[0149] With the user having selected the voice text, the user performs an operation to request the 3D image information and captured image of the property (by pressing the image acquisition button 213). The user may request the 3D image information and captured image simply by selecting the voice text. The input reception unit 12 of the terminal device 10 receives the operation to request the 3D image information and captured image of the property. The 3D image information of the property is the 3D image information of the items placed in the property, which has been generated as a virtual space. The items are composed of 3D model shape information.

[0150] Here, the voice-to-text display screen 210 includes a first display area 214 that displays voice-to-text acquired from the conference management server 20, and a second display area 215 that displays three-dimensional image information and captured images of items acquired from the image management server 40. At step S17, the voice-to-text is displayed in the first display area 214, but nothing is displayed in the second display area 215.

[0151] S18: If the user is not logged in to the image management server 40, the user enters a login operation into the terminal device 10. This login operation is directed to the image management server 40. The input reception unit 12 of the terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful. Alternatively, the user's login operation may be omitted by using, for example, single sign-on.

[0152] S19: The terminal device 10 executes an image request program to request three-dimensional image information. The transmitting / receiving unit 11 then sends a request for three-dimensional image information and a captured image of the property selected by the user to the image management server 40, specifying the property identification information and the date and time of sound collection. Since the captured image is acquired in conjunction with the three-dimensional image information, the captured image is omitted in Figure 10. The transmitting / receiving unit 11 should also send the URL of the conference management server 20 to the image management server 40 so that the terminal device 10 can be redirected to the conference management server 20. The three-dimensional image information of the property is an image of an object placed in the property, which is formed as a virtual space. Since the object is composed of 3D model shape information, the terminal device 10 projects the three-dimensional model shape of the object onto two dimensions to generate a planar image. The user can view any object while changing their viewpoint. The transmitting / receiving unit 11 may also have the image management server 40 receive property management information acquired from the conference management server 20. The image request program receives property management information from a web application connected to the conference management server 20, for example, as a URL parameter.

[0153] S20: The transmitting / receiving unit 41 of the image management server 40 receives requests for three-dimensional image information and captured images of the object. The storage / reading unit 49 searches the three-dimensional image information management DB 4001 using the object identification information and obtains the three-dimensional image information for each item. The storage / reading unit 49 also searches the captured image information management DB 4006 using the object identification information and obtains the captured image (an example of a two-dimensional image), location information, and field of view information that corresponds to the shooting date and time closest to the sound collection date and time. The processing unit 47 requests the screen generation unit 42 to generate a screen that includes the three-dimensional image information and captured images of the object. The screen generation unit 42 generates the three-dimensional image information by placing a virtual camera at the location information and determining the field of view of the virtual camera based on the field of view information. The screen generation unit 42 generates a screen corresponding to a second display area 215 in which the three-dimensional image information and captured images of each item are arranged on one screen.

[0154] The transmitting / receiving unit 41 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The three-dimensional image information of each item included in the screen information is image information of all items included in the property placed on the property, and the user can change the viewpoint as they wish.

[0155] S21: The transmitting / receiving unit 11 of the terminal device 10 receives screen information for the screen corresponding to the second display area 215, and the display control unit 13 displays a text image display screen 220 including the first display area 214 and the second display area 215 (see Figure 13). In step S11, the second display area 215 displays three-dimensional image information and captured images of each item. The first display area 214 displays the audio text. Therefore, the audio text with the specified recording date and time, and the captured image from the date and time closest to this recording date and time, are displayed on a single screen together with the three-dimensional image information.

[0156] The user can change the viewpoint of the three-dimensional image information or enlarge the object. The user performs an operation to request past information. Past information refers to captured images prior to the captured image displayed in step S21 (hereinafter referred to as past captured images) and audio text prior to the selected audio text. The operation to request past information may be, for example, pressing the information display button 225. The input reception unit 12 of the terminal device 10 receives the operation to request past information. The user may specify a specific past date and time. In addition, although past information is requested in this embodiment, the user may also request information in the future than the audio text selected in step S17.

[0157] S22: When the user presses the information display button 225, the transmitting / receiving unit 11 of the terminal device 10 requests past information from the image management server 40 by specifying the field of view information. This field of view information is the field of view specified by the user for the three-dimensional image information. This is how past information is requested. In addition, the coordinates pressed by the user with the mouse pointer or the model ID of the item identified by the coordinates is sent to the image management server 40.

[0158] S23: The transmitting / receiving unit 41 of the image management server 40 receives a request for past information. The storage / reading unit 49 identifies the closest field of view information to the received field of view information from the captured image information management DB 4006, which is older than the sound collection date and time in step S19. The storage / reading unit 49 obtains the shooting date and time associated with the identified field of view information. It is not guaranteed that field of view information exactly the same as the received field of view information will be found in the captured image information management DB 4006. For this reason, the storage / reading unit 49 identifies field of view information from the captured image information management DB 4006 that has only a difference of a certain amount in the field of view. The extent to which the data should be traced back in time may also be pre-set. If multiple field of view information matches, the storage / reading unit 49 identifies the most recent image information. The user may be able to set the difference limit and the extent to which the data should be traced back in time.

[0159] Furthermore, the memory / reading unit 49 retrieves the captured image associated with the date and time of shooting from the captured image information management DB 4006 (hereinafter referred to as past captured image).

[0160] The transmitting / receiving unit 41 of the image management server 40 transmits the date and time of shooting, which is identified by the field of view information, to the terminal device 10.

[0161] S24: The transmitting / receiving unit 11 of the terminal device 10 receives the shooting date and time identified by the field of view information. For example, the image management server 40 notifies the terminal device 10 of the URL of the conference management server 20 and redirects it. As a result, the transmitting / receiving unit 11 of the terminal device 10 sends a voice-to-text request to the conference management server 20, specifying the shooting date and time identified by the field of view information.

[0162] S25: The transmitting / receiving unit 21 of the conference management server 20 receives a request for voice text, and the storage / reading unit 29 searches the conference information management DB 2001 for the recording date and time using the received recording date and time. The storage / reading unit 29 retrieves the voice text (recording device) and voice text (communication terminal) for the current date and time from the conference information management DB 2001 that correspond to the same or the closest recording date and time as the recording date and time. Hereinafter, these voice texts will be referred to as past texts. The transmitting / receiving unit 21 transmits the retrieved past texts to the terminal device 10.

[0163] S26: When the transmitting / receiving unit 11 of the terminal device 10 receives past text, it sends the past text to the image management server 40. The transmitting / receiving unit 41 of the image management server 40 receives the past text as a response to the request in step S23. The storage / reading unit 49 stores the past text in the captured image information management DB 4006, associating it with the shooting date and time identified in step S23. This associates the audio text with the captured image.

[0164] S27: When past text is received, the processing unit 47 associates the captured image displayed in step S20, the three-dimensional image information of the field of view received in step S22, the past captured image identified in step S23, and the past text, and requests the screen generation unit 42 to display them. The screen generation unit 42 generates three-dimensional image information by determining the field of view of the virtual camera based on the field of view information in step S22. The screen generation unit 42 associates the generated three-dimensional image information, the captured image in step S20, the past captured image, and the past text, and generates a screen corresponding to the second display area 215. The transmitting / receiving unit 41 transmits the screen information of the screen corresponding to the second display area 215 to the terminal device 10.

[0165] S28: The transmitting / receiving unit 11 of the terminal device 10 receives image information of the screen information corresponding to the second display area 215, and the display control unit 13 displays the past text and past capture display screen 230 including the first display area 214 and the second display area 215 (see Figure 14). In step S28, the first display area 214 displays the audio text as in step S11, and the second display area 215 displays the three-dimensional image information of the item, the captured image displayed in step S20, the past captured image, and the past text.

[0166] <Screen example> Figure 11 shows an example of a property selection screen 200 for entering property identification information. The property selection screen 200 has a property identification information input field 201 and a search button 202. When the user enters property identification information in the property identification information input field 201 and presses the search button 202, the list of room numbers shown in Figure 12 is displayed on the voice-text display screen 210.

[0167] Figure 12 shows an example of a voice-to-text display screen 210. The voice-to-text display screen 210 includes a first display area 214 that displays voice-to-text acquired from the conference management server 20, and a second display area 215 that displays image information of items acquired from the image management server 40. The first display area 214 is the area other than the second display area 215. The first display area 214 includes voice-to-text acquired in the conference of an item identified by the item identification information. The user selects the voice-to-text 217 related to the item for which they want to display a captured image using the mouse cursor 212. The date and time of sound collection 216 is also identified by the selection of the voice-to-text. When the user presses the image acquisition button 213, the text image display screen 220 is displayed.

[0168] The second display area 215 is an area other than the first display area 214, and its display may be implemented by a program on a web application, such as an iframe.

[0169] Figure 13 shows an example of a text-image display screen 220. The text-image display screen 220 includes a first display area 214 and a second display area 215, and includes audio text, captured images, and three-dimensional image information on a single screen. The first display area 214 is the same as in Figure 12.

[0170] The second display area 215 of the text image display screen 220 displays the captured image 237 and the three-dimensional image information 222. The captured image 237 is the image taken at the date and time closest to the sound collection date and time 216 associated with the selected audio text 217 (held by the image management server 40). Initially, the three-dimensional image information 222 is taken from the same shooting position and field of view as the captured image 237, but since the three-dimensional image information 222 is a wide-field image, the user can change the field of view information.

[0171] Furthermore, if a user wants to view past captured images of any item, they can manipulate the three-dimensional image information 222 to specify the field of view for displaying the desired item (viewpoint). For example, the user specified a field of view to zoom in on a table. When the user then presses the information display button 225, the past text and past capture display screen 230 is displayed. In addition to displaying past text and past captures, the information display button 225 is also used to display text information generated based on tacit knowledge comments, as will be described later. Furthermore, when the user presses the information update button 226, the tacit knowledge model 4004 is updated.

[0172] In Figure 13, the area (floor area) 224 is displayed as information about the property. The area (floor area) 224 may be a measured value or may be included in the three-dimensional image information management table.

[0173] Figure 14 shows an example of a past text past capture display screen 230. The past text past capture display screen 230 includes a first display area 214 and a second display area 215. The first display area 214 displays the audio text.

[0174] In Figure 14, three-dimensional image information 223 of a table selected by the user is displayed in the second display area 215 as an example of an item. The field of view of the three-dimensional image information 223 is the field of view specified by the user to view the table. In other words, the user manipulated the field of view so that the table appeared in this way. The second display area 215 also displays the captured image 237, as in Figure 13. The second display area 215 also displays a past captured image 238. The past captured image 238 is the most recent captured image from before the captured image 237. Multiple past captured images may be displayed in the second display area 215. The past captured image 238 is the captured image with the field of view information closest to the field of view of the three-dimensional image information 223 specified by the user, and therefore it is a captured image with the same or a similar field of view as the three-dimensional image information 223.

[0175] In this way, the user can select the audio text 217 to identify the recording date and time 216, and display the captured image 237 associated with the recording date and time 216. Furthermore, the user can specify the field of view in the three-dimensional image information 223 to display a captured image 238 that is earlier than the captured image 237. By matching the field of view of the captured image 237 when specifying the field of view, the user can display objects from different dates and times at the same field of view. Therefore, the user can compare a three-dimensional model of an object with multiple captured images taken at different dates and times from the same field of view.

[0176] In addition, in the second display area 215 of Figure 14, only one of the three-dimensional image information 223 or the previously captured image 238 may be displayed. Furthermore, the captured image 237 may not be displayed.

[0177] Furthermore, the second display area 215 displays past text 232. An example of past text 232 is "This is the initial state." Since past text 232 is audio text identified based on the date and time the past capture image 238 was taken, it can be expected to be comment data related to the past capture image 238.

[0178] In this way, the terminal device 10 can display voice text, three-dimensional image information 223, captured image 237, past captured image 238, and past text 232 on a single screen.

[0179] As shown in Figure 15, if the user selects a different text, the information displayed in the second display area 215 will also correspond to that text. Figure 15 shows the past text and past capture display screen 230 when the user selects a different voice text 218. In Figure 15, the user selected voice text 218. Therefore, the second display area 215 displays the three-dimensional image information 227, the captured image 243, and the past captured image 244 of the prism-shaped object. The captured image 243 is identified by the date and time the voice text 218 was collected, and the past captured image 244 is identified by the field of view information for the three-dimensional image information 227. In addition, the second display area 215 displays the past text 228, identified by the date and time the past captured image 244 was taken.

[0180] In this way, by selecting the audio text 218, the user can switch the images or text displayed in the second display area 215.

[0181] <Example of an image management server retrieving inspection information from a conference management server> In Figure 10, the terminal device 10 retrieves past text from the conference management server 20, and the image management server 40 retrieves it from the terminal device 10. However, it is also possible for the image management server 40 to directly retrieve past text from the conference management server 20.

[0182] Figure 16 is a sequence diagram (modified) showing an example of a process that generates screen information by arranging both audio text and captured images or three-dimensional image information. Note that Figure 16 may mainly explain the differences from Figure 10. First, the processing in steps S11 to S22 can be the same as in Figure 11.

[0183] S23: The transmitting / receiving unit 41 of the image management server 40 receives a request for past information along with the field of view information. The processing is the same as in Figure 10. The transmitting / receiving unit 41 of the image management server 40 specifies the shooting date and time identified by the field of view information and requests the conference management server 20 for the audio text of the shooting date and time. The storage / reading unit 49 also retrieves the past captured image associated with this shooting date and time.

[0184] The transmitting / receiving unit 21 of the conference management server 20 receives a request for voice-to-text. The processing is the same as in Figure 10. The storage / retrieval unit 29 of the conference management server 20 retrieves the voice-to-text for the current date and time (camera) and the voice-to-text for the current date and time (communication terminal) from the conference information management DB 2001 based on the received date and time of capture. The transmitting / receiving unit 21 then transmits the retrieved past text to the image management server 40.

[0185] S26: The transmitting / receiving unit 41 of the image management server 40 receives past text as a response to the request in step S23.

[0186] The subsequent processing can be the same as in Figure 11. With processing as shown in Figure 16, the processing required for the terminal device 10 to change the connection destination can be reduced, thus shortening the time until the past text and past capture display screen 230 is displayed.

[0187] <Main effects, etc.> In this embodiment, the image management server 40 performs processing based on at least one of three-dimensional image information or past captured images and past text, so that the terminal device 10 can display at least one of the three-dimensional image information or past captured images and past text in the second display area 215. The image management server 40 can perform processing based on past text managed by the conference management server 20 and at least one of the three-dimensional image information or past captured images managed by the image management server 40 without adding processing functions to the conference management server 20.

[0188] Furthermore, the conference management server 20 may perform only some of the processing based on three-dimensional image information or past captured images and past text. Even in this case, the processing load on the conference management server 20 is reduced compared to performing all of the processing based on three-dimensional image information or past captured images and past text on the conference management server 20 side.

[0189] Here, the three-dimensional image information or at least one of the previously captured images and the audio text may be displayed in a manner that does not overlap with each other, or they may be displayed in a manner that overlaps with each other. Furthermore, the first display area 214 and the second display area 215 may be displayed so as not to overlap with each other, or they may be displayed so as to overlap with each other. Furthermore, the first display area 214 and the second display area 215 may each be divided into multiple sections for display, and the divided sections may be displayed in a mixed manner.

[0190] [Second Embodiment] In this embodiment, we will describe an image management server 40 that acquires tacit knowledge comments from a tacit knowledge model using at least one of three-dimensional image information and captured images, and generates text information based on the tacit knowledge comments.

[0191] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 2 and the functional block diagram in Figure 3, as described in the above embodiment, can be used as reference.

[0192] <Action or process> <<Learning Phase (Model Update)>> First, referring to Figure 17, we will explain the model update process in which the tacit knowledge model 4004 learns data. Figure 17 is a sequence diagram showing an example of the model update process. In explaining Figure 17, we will mainly explain the differences from Figure 10. The process in steps S31 to S40 can be the same as in Figure 10.

[0193] S41: In step S41, in addition to the processing in step S21, the user inputs comments (text information, audio) about the item into the terminal device 10, as explained in Figures 9 and 10. Comments are sometimes referred to as input information. This input information may be tacit knowledge comments. The input information may also include caption comments that describe the item.

[0194] S42: When the user presses the information update button 226, the transmitting / receiving unit 11 of the terminal device 10 sends a request for past information (angle of view information, input information) to the image management server 40.

[0195] The subsequent steps S43 to S46 can be the same as in Figure 11.

[0196] S47: The transmitting / receiving unit 41 of the image management server 40 receives past text. The determination unit 43 obtains a caption comment identified by the model ID (transmitted in step S42) from the caption model 4003 and determines the degree of relevance between the caption comment and the comments included in the input information received in step S42. The determination unit 43 may determine the degree of relevance between the obtained caption comment and all comments included in the input information received in step S42, or it may divide the comments included in the input information received in step S42 into multiple parts and determine the degree of relevance between the obtained caption comment and each divided comment.

[0197] S48: The update unit 46 updates the caption model 4003 by associating the input information deemed highly relevant in step S47 with the model ID as a caption comment. The update unit 46 also updates the tacit knowledge model 4004 using the input information and past text deemed less relevant in step S47, along with the three-dimensional image information (with the field of view in step S42 as a predetermined region image) and past captured images as training data. In other words, the correspondence between the three-dimensional image information and past captured images of the items and the past text and input information is learned. Features are extracted from the three-dimensional image information and past captured images of the items using several feature extraction models suitable for images, such as CNNs. Features represent, for example, what is pictured where, or the content of the work being performed. Therefore, the tacit knowledge model 4004 can learn the correspondence between the features of the three-dimensional image information and past captured images of the items and the past text and input information.

[0198] The update unit 46 does not need to use three-dimensional image information of the item or past captured images to update the tacit knowledge model 4004.

[0199] In addition, in the second display area 215 of Figure 14, only one of the three-dimensional image information 223 or the previously captured image 238 may be displayed. Furthermore, the captured image 237 may not be displayed.

[0200] Note that both past text and input information are not necessarily required; the tacit knowledge model 4004 can be updated with at least one of them.

[0201] Furthermore, the update unit 46 may also use the audio text selected in step S37 and the captured image taken at the closest time to the audio text's recording date and time for training. However, since these recording dates and times are not necessarily close, the update unit 46 may only use them for training if the difference between the recording date and time and the shooting date and time is within a certain time frame.

[0202] Furthermore, while Figure 17 shows the image management server 40 acquiring past text from the terminal device 10, it may also be acquired from the conference management server 20, as shown in Figure 16.

[0203] <<Example of a screen during the learning phase>> The screen displayed by terminal device 10 during the learning phase will be the same as in Figures 11 to 13, but in Figure 13, the user can input information. In addition, the information corresponding to Figure 14, which displays past captured images and past text, will be displayed in the inference phase described below.

[0204] Figure 18 shows an example of a text image display screen 220 displayed by the terminal device 10. The text image display screen 220 in Figure 18 includes a first display area 214 and a second display area 215. The first display area 214 displays the area (floor area) 224 as information about the property. The second display area 215 displays a captured image 237 and three-dimensional image information 223. In addition, input information 241 entered by the user, "This table has an unstable center of gravity, so it is best not to place anything weighing more than 50 kg on it," is displayed.

[0205] The image management server 40 can update the tacit knowledge model 4004 using this input information 241. Note that the area (floor area) 224, as information about the property, can be used as a caption comment.

[0206] <Inference Phase (Generation of Text Information)> Next, with reference to Figure 19, the text information generation process using the tacit knowledge model 4004 will be explained. Figure 19 is a sequence diagram illustrating an example of the text information generation process. Note that the explanation of Figure 19 will mainly focus on the differences from Figure 10. The processes in steps S31 to S46 are the same as in Figure 10. However, in step S41, the user inputs a question about an item (see Figure 20).

[0207] S51: The transmitting / receiving unit 41 of the image management server 40 receives past text. The processing unit 47 requests the text information generation unit 45 to generate text information. The text information generation unit 45 obtains tacit knowledge comments corresponding to the three-dimensional image information of the item and the past captured image identified in step S42 from the tacit knowledge model 4004. The tacit knowledge model 4004 can extract features of the three-dimensional image information of the item and the past captured image, and identify at least one of the past text and input information corresponding to these features. The tacit knowledge model 4004 extracts at least one of the past text and input information as tacit knowledge comments.

[0208] S52: Next, the text information generation unit 45 obtains text information created by the large-scale language model using the tacit knowledge comments, input information (question text), and past text. The large-scale language model 4005 can use these to generate more detailed text information. The text information generation unit 45 may also convert the audio information contained in the input information (question text) into text information, and the text information generated by the text information generation unit 45 may be either audio information or text information.

[0209] The text information generation unit 45 may generate text information without using any past text or input information (question text). Alternatively, the text information generation unit 45 may generate fixed questions internally within the system and use these fixed questions. In this case, the question text is not visible to the user. Or, the text information generation unit 45 may generate fixed questions internally within the system, display these fixed questions on the display unit for the user to select, and then use the selected question.

[0210] While past text and input information are not mandatory, generating text information from the large-scale language model 4005 using past text and input information allows for more detailed information about the item. For example, if past text or input information includes the extent of damage to the item, text information can be generated that includes appropriate responses based on the extent of the damage.

[0211] S53: The processing unit 47 associates the captured image displayed in step S40, the three-dimensional image information of the field of view received in step S42, the previously identified captured image in step S43, and the text information, and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the three-dimensional image information, the captured image, the previously identified captured image, and the generated text information.

[0212] The screen generation unit 42 may perform an update process that adds only text information to the screen corresponding to the second display area 215. The transmitting / receiving unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmitting / receiving unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0213] S54: The display control unit 13 of the terminal device 10 displays a past text past capture display screen 230 including a first display area 214 and a second display area 215 (see Figure 21). Alternatively, the conversion unit 15 converts the received text information into audio information, and the audio control unit 14 plays the converted text information to the speaker 109a. If the received text information is audio information, the text information is played to the speaker 109a, or the conversion unit 15 converts the received text information into character information and displays the converted text information on the display 106a.

[0214] In Figure 19, the image management server 40 retrieves past text from the terminal device 10, but it may also be retrieved from the conference management server 20 as shown in Figure 16.

[0215] <<Example of the inference phase screen>> The screen displayed by the terminal device 10 during the inference phase will be the same as in Figures 11 to 13, but in Figure 13, the user can input a question.

[0216] Figure 20 shows an example of a text-image display screen 220 in the inference phase. The text-image display screen 220 includes a first display area 214 and a second display area 215. Figure 20 has a similar configuration to Figure 13, but the user has entered a question as input information. The second display area 215 displays three-dimensional image information 222, a captured image 237 identified by the selection of the audio text 217, and input information (question) 234. For example, the input information (question) 234 in Figure 20 is "There is a scratch on the table, what should I do?" Along with such input information 234, the user presses the information display button 225 to request the generation of text information using the tacit knowledge model.

[0217] Figure 21 shows an example of text information displayed on the Past Text Past Capture Display Screen 230. The Past Text Past Capture Display Screen 230 includes a first display area 214 and a second display area 215. The second display area 215 displays the three-dimensional image information 223 of the table, the captured image 237 based on the audio text 217, the past captured image 238, and the text information 235. This text information 235 is "The scratch is less than 1 mm deep, so it will be repaired with paint. If it is 1 mm or more, it will be polished." The tacit knowledge model 4004 generates a tacit knowledge comment based on the three-dimensional image information 223 of the item and the past captured image 238. The large-scale language model 4005 generates the text information 235 from the tacit knowledge comment, past text, and input information (question sentence).

[0218] For example, if a scratch is detected in past captured image 238, an implicit knowledge comment regarding the scratch in the table is extracted. This implicit knowledge comment, a question about the scratch, and past text about the scratch are input into the large-scale language model 4005, which can then generate appropriate text information corresponding to the scratch in the table.

[0219] Text information 235 can be described as the result of processing based on past text and three-dimensional image information or past captured images.

[0220] [Third Embodiment] In this embodiment, an image management server 40 that generates images from captured images and text information will be described.

[0221] Figure 22 is a functional configuration diagram illustrating an example of the functions of the image management server 40, the conference management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment. Note that the explanation of Figure 22 mainly focuses on the differences from Figure 3.

[0222] The image management server 40 in Figure 22 further includes an image generation unit 48, and an image generation model 4007 is constructed in the storage unit 4000 of the image management server 40. The other configurations may be the same as in Figure 3.

[0223] The image generation unit 48 is an example of an image generation means, and is implemented by instructions from the CPU 401 shown in Figure 2. It inputs text data, or text data and images, into the image generation model 4007 to generate image information.

[0224] Image generation model 4007 is a machine learning model (generative AI) that generates images from text data, or from text data and images. Image generation model 4007 is trained using training data that includes text data and images, for example. This training data includes, for example, training text data, or text data and images as input, and ground truth images as output. For example, the image generation model 4007 may be trained so that the images it generates when inputting text data, or text data and images, from the training data approach the ground truth images contained in the training data.

[0225] <Learning Phase> The processing in the learning phase may be the same as in Figure 17. In step S48, the update unit 46 updates the tacit knowledge model 4004 to learn a correspondence between comments and past text that were determined to have low relevance in step S47, and three-dimensional image information or past captured images of the item. Alternatively, the update unit 46 updates the tacit knowledge model 4004 to learn a correspondence between comments, past text, and three-dimensional image information (or captured images) of the item, and past captured images (or three-dimensional image information) of the item.

[0226] <Inference Phase (Generation of Text Information)> Figure 23 is an example sequence diagram showing the process of generating text and image information. Note that the explanation of Figure 23 mainly focuses on the differences from Figure 19. In Figure 23, step S52-1 is added.

[0227] S52-1: The image generation unit 48 inputs previously captured images and text information created by the large-scale language model to the image generation model 4007 to generate image information. The image generation unit 48 may also obtain image information created by the image generation model 4007 using text information created by the large-scale language model without using previously captured images.

[0228] The memory / reading unit 49 stores (or overwrites) the text information created by the large-scale language model and the image information created by the image generation model 4006 in the three-dimensional image information management DB 4001, corresponding to the past text stored in the three-dimensional image information management DB 4001 in step S46.

[0229] S53: The processing unit 47 associates the three-dimensional image information of the item corresponding to the model identification information with the generated image information and text information, and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the three-dimensional image information of the item, the generated image information, and text information corresponding to the three-dimensional image information of the item and the captured image. The transmitting / receiving unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0230] <Example of the inference phase screen> Figure 24 shows the generated image information displayed on the text image display screen 260. The explanation of Figure 24 mainly focuses on the differences from Figure 21.

[0231] The text image display screen 260 in Figure 24 shows generated images 261 and 262. Generated images 261 and 262 are not the captured image 237 and past captured image 238 described in Figure 21, but rather generated images produced by the image generation model 4007 based on the past captured image 238 and text information 235. Therefore, generated images 261 and 262 have markers 263 and 264 that indicate the location of the damage. Note that one of the generated images 261 or 262 may be the past captured image 238, or it may be possible to switch between them through user operation.

[0232] <Effects of generating text information using captured images> This invention will explain the effects of generating text information using captured images.

[0233] 1. Comparative Example 1 (Using a General Large-Scale Language Model) Question: The user asks, "How should I repair the cracks?" Tacit comment: Tape or filler can be used.

[0234] 2. Comparative Example 2 (Case of learning from three-dimensional image information) • Learning Phase Training data: Users display a 3D image and ask, "How should I repair the cracks?" Input information: Use tape for wide cracks and filler for narrow cracks. • Inference Phase Input image: Three-dimensional image information Question: "How should I repair the cracks?" Implicit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter.

[0235] 3. This embodiment (three-dimensional image information, past captured images, and past text) • Learning Phase Input images: Three-dimensional image information and past captured images Previous text: Applying tape to corners may cause cracking. • Inference Phase Input images: Three-dimensional image information and past captured images Question: "How should I repair the cracks?" Implicit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter, but be careful when applying tape to corners as cracks may occur. In other words, "Be careful when applying tape to corners as cracks may occur" is an effect of having studied the previous text.

[0236] 4. This embodiment (three-dimensional image information, past captured images, past text, input information) • Learning Phase Input images: 3D images and past captured images Previous text: Applying tape to corners may cause cracking. Input information: The wide cracks span across the corners. • Inference Phase Input images: Three-dimensional image information and past captured images Question: "How should I repair the cracks?" Tacit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter, but be careful when applying tape to corners as it may cause cracks to form. In other words, the effect of having studied the previous text is that "be careful when applying tape to corners as it may cause cracks to form."

[0237] <Multimodal> Several examples of combinations of input information and tacit knowledge comments are described. While the model described above assumed a large-scale language model, this embodiment can utilize a multimodal model that takes multiple data formats (images, text, gestures, etc.) as input and outputs them in a predetermined data format. • When the input information is a string, and content other than text information is generated as an implicit knowledge comment. Enter text to generate an image Enter text to generate a video. Enter text to generate speech. Enter a string of text to generate a 3D model. • When input information includes both strings and non-strings, and text information is generated as an implicit knowledge comment. Enter an image and text to generate text information. Enter a 3D model and text to generate text information. Input speech and text to generate text information. • When input information includes both text and non-textual content, and non-textual content is generated as an implicit knowledge comment. Enter an image and text to generate an image. Enter a video and text to generate a video. Enter a 3D model and text to generate a 3D model. Input speech and text to generate speech. <Main effects> In this embodiment, the image management server 40 updates the tacit knowledge model using the three-dimensional image information, captured images, and audio text as processing based on at least one of the three-dimensional image information and past captured images and past text, so that the terminal device 10 can display tacit knowledge comments corresponding to at least one of the three-dimensional image information and past captured images.

[0238] [Other application examples] The present invention is not limited to the embodiments specifically disclosed above, and various modifications and changes are possible without departing from the scope of the claims. It goes without saying that the image management server 40 described in this embodiment is just one example, and there are various system configurations depending on the application and purpose.

[0239] For example, in this embodiment, we described an example in which an tacit knowledge model for industries such as civil engineering and construction answers a question, but the tacit knowledge model can be used in any industry where tacit knowledge is effective, such as healthcare, dental care, and investment decisions.

[0240] Furthermore, in this embodiment, the large-scale language model 4005 generates text information based on tacit knowledge comments, but it is also possible to use tacit knowledge comments as text information without using the large-scale language model 4005.

[0241] Furthermore, the tacit knowledge model 4004 may also be trained to take three-dimensional image information and audio text as input, and output tacit knowledge comments using the input information. In other words, the tacit knowledge model 4004 may take information in different forms, such as images and text, as input.

[0242] Furthermore, although this embodiment describes a client-server type information processing system 100, the functions of the image management server 40 may be installed as an application on the terminal device 10. In other words, users may use the functions of this embodiment in a standalone format.

[0243] Furthermore, the configuration examples shown in Figure 3 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the image management server 40. The present invention is not limited by the way the processing units are divided or their names. The processing of the image management server 40 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processing.

[0244] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0245] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the image management server 40 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0246] Furthermore, the image management server 40 can combine the disclosed processing steps in various ways. Each element of the image management server 40 may be combined into a single device or divided into multiple devices. Also, each processing performed by the image management server 40 may be performed by the terminal device 10.

[0247] <Mode> [Aspect 1] An information processing system comprising: a first server that manages text data based on audio data acquired when an image of an object is captured by a shooting device; a second server that manages three-dimensional image information of the object and the captured image aligned with the three-dimensional image; and a terminal device capable of communicating with the first server and the second server, The aforementioned terminal device is A display control unit that displays a display screen including the text data received from the first server and the three-dimensional image information received from the second server is provided. The second server identifies the captured image based on the viewing angle of the three-dimensional image information received by the terminal device, obtains the captured image from the first server, and associates the text data with the three-dimensional image information of the viewing angle, or associates the generated information generated based on the text data with the three-dimensional image information of the viewing angle, and includes a processing unit that executes the processing. The display control unit of the terminal device displays the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information received from the second server on the display screen. An information processing system. [Aspect 2] The display control unit of the terminal device displays a first display area for displaying the text data received from the first server, and displays a second display area for displaying the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information received from the second server, and displays the display screen including the same. The information processing system according to Aspect 1. [Aspect 3] The second server stores the text data transmitted by the first server in association with the captured image in a storage unit. The information processing system according to Aspect 1 or 2. [Aspect 4] The processing unit requests the first server for the text data and obtains the text data from the first server as a response to the request. The information processing system according to Aspect 1. [Aspect 5] The second server The system has a model that has learned the correspondence between the captured image identified based on the field of view of the three-dimensional image information received by the terminal device, the three-dimensional image information of the field of view received by the terminal device, and the text data transmitted by the first server. The aforementioned processing unit, Based on the captured image and the three-dimensional image information of the field of view received by the terminal device, the text data generated by the model is acquired as the generated information. An information processing system as described in any one of the four descriptions (1 to 4). [Aspect 6] The second server described above is The system has a model that learns the correspondence between the captured image identified based on the field of view of the three-dimensional image information received by the terminal device, the three-dimensional image information of the field of view received by the terminal device, the text data transmitted by the first server, and the input information received from the terminal device. The aforementioned processing unit, Based on the captured image and the three-dimensional image information of the field of view received by the terminal device, the text data generated by the model is acquired as the generated information. An information processing system as described in any one of the four descriptions (1 to 4). [Aspect 7] The second server described above is The update unit has an update unit that updates the model by learning the correspondence between the captured image identified based on the field of view of the three-dimensional image information received by the terminal device, the three-dimensional image information of the field of view received by the terminal device, and the text data transmitted by the first server. The information processing system described in aspect 5. [Aspect 8] The second server described above is The update unit has an update unit that updates the model by learning the correspondence between the captured image and the three-dimensional image information of the field of view received by the terminal device, which is identified based on the field of view of the three-dimensional image information received by the terminal device, and the text data transmitted by the first server and the input information received from the terminal device. The information processing system described in Embodiment 6. [Aspect 9] The aforementioned processing unit, The text data obtained from the first server is associated with the captured image identified based on the field of view of the three-dimensional image information received by the terminal device, or The process of associating the generated information with the captured image identified based on the field of view of the three-dimensional image information received by the terminal device is performed. The display control unit of the terminal device displays the associated text data and the captured image, or the associated generated information and the captured image, received from the second server, on the display screen. An information processing system as described in any one of the eight descriptions (1 to 8). [Explanation of Symbols]

[0248] 10 Terminal devices 20 Conference Management Server 40 Image Management Server 100 Information Processing Systems [Prior art documents] [Patent Documents]

[0249] [Patent Document 1] Patent No. 6954410

Claims

1. An information processing system comprising: a first server that manages text data based on audio data acquired when an image of an object is captured by a shooting device; a second server that manages three-dimensional image information of the object and the captured image aligned with the three-dimensional image information; and a terminal device capable of communicating with the first server and the second server, The aforementioned terminal device is The system includes a display control unit that displays a display screen containing the text data received from the first server and the three-dimensional image information received from the second server, The second server is, The system includes a processing unit that identifies the captured image based on the field of view of the three-dimensional image information selected by the terminal device, retrieves the text data acquired when the captured image was taken from the first server, and performs a process to associate the text data with the three-dimensional image information of the field of view, or to associate generated information generated based on the text data with the three-dimensional image information of the field of view. The display control unit of the terminal device displays the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information, received from the second server, on the display screen. Information processing system.

2. The display control unit of the terminal device is A first display area for displaying the text data received from the first server, The display screen includes a second display area that displays the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information, received from the second server. The information processing system according to claim 1.

3. The second server is, The text data transmitted by the first server is stored in the storage unit in association with the captured image. The information processing system according to claim 1 or 2.

4. The processing unit requests the text data from the first server and obtains the text data from the first server as a response to the request. The information processing system according to claim 1.

5. The second server is, The system has a model that has learned the correspondence between the captured image identified based on the field of view of the three-dimensional image information received by the terminal device, the three-dimensional image information of the field of view received by the terminal device, and the text data transmitted by the first server. The aforementioned processing unit, Based on the captured image and the three-dimensional image information of the field of view received by the terminal device, the text data generated by the model is acquired as the generated information. The information processing system according to claim 1.

6. The second server is, The system has a model that learns the correspondence between the captured image identified based on the field of view of the three-dimensional image information received by the terminal device, the three-dimensional image information of the field of view received by the terminal device, the text data transmitted by the first server, and the input information received from the terminal device. The aforementioned processing unit, Based on the captured image and the three-dimensional image information of the field of view received by the terminal device, the text data generated by the model is acquired as the generated information. The information processing system according to claim 1.

7. The second server is, The update unit has an update unit that updates the model by learning the correspondence between the captured image identified based on the field of view of the three-dimensional image information received by the terminal device, the three-dimensional image information of the field of view received by the terminal device, and the text data transmitted by the first server. The information processing system according to claim 5.

8. The second server is, The update unit has an update unit that updates the model by learning the correspondence between the captured image and the three-dimensional image information of the angle of view received by the terminal device, which is identified based on the angle of view of the three-dimensional image information received by the terminal device, and the text data transmitted by the first server and the input information received from the terminal device. The information processing system according to claim 6.

9. Claim 1 also displays a captured image in addition to the three-dimensional image information. The aforementioned processing unit, The text data obtained from the first server is associated with the captured image identified based on the field of view of the three-dimensional image information received by the terminal device, or The process of associating the generated information with the captured image identified based on the field of view of the three-dimensional image information received by the terminal device is performed. The display control unit of the terminal device displays the associated text data and the captured image, or the associated generated information and the captured image, received from the second server, on the display screen. The information processing system according to claim 1.

10. The second server in an information processing system includes: a first server that manages text data based on audio data acquired when an image of an object is captured by a shooting device; a second server that manages three-dimensional image information of the object and the captured image aligned with the three-dimensional image information; and a terminal device that can communicate with the first server and the second server and displays a display screen including the text data received from the first server and the three-dimensional image information received from the second server, The system includes a processing unit that identifies the captured image based on the field of view of the three-dimensional image information selected by the terminal device, retrieves the text data acquired when the captured image was taken from the first server, and performs a process to associate the text data with the three-dimensional image information of the field of view, or to associate generated information generated based on the text data with the three-dimensional image information of the field of view. The terminal device is configured to display on the display screen the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information, which have been received from the second server. The second server.

11. The first server in an information processing system includes: a first server that manages text data based on audio data acquired when an image of an object is captured by a shooting device; a second server that manages three-dimensional image information of the object and the captured image aligned with the three-dimensional image information; and a terminal device that can communicate with the first server and the second server and displays a display screen including the text data received from the first server and the three-dimensional image information received from the second server, The second server identifies the captured image based on the field of view of the three-dimensional image information selected by the terminal device, obtains the text data acquired when the captured image was taken from the first server, associates the text data with the three-dimensional image information of the field of view, or associates the generated information generated based on the text data with the three-dimensional image information of the field of view. To display the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information, on the display screen of the terminal device, A transmission unit that transmits the text data to the second server. A first server having [a certain feature].

12. An information processing method performed by the second server in an information processing system, the system comprising: a first server that manages text data based on audio data acquired when an image of an object is captured by a shooting device; a second server that manages three-dimensional image information of the object and the captured image aligned with the three-dimensional image information; and a terminal device that can communicate with the first server and the second server and displays a display screen including the text data received from the first server and the three-dimensional image information received from the second server, wherein the second server performs information processing, The terminal device identifies the captured image based on the field of view of the three-dimensional image information selected by the terminal device, retrieves the text data acquired when the captured image was taken from the first server, associates the text data with the three-dimensional image information of the field of view, or associates generated information generated based on the text data with the three-dimensional image information of the field of view. The terminal device performs a process to display the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information, received from the second server, on the display screen. An information processing method that performs this task.

13. An information processing system comprising: a first server that manages text data based on audio data acquired when an image of an object is captured by a shooting device; a second server that manages three-dimensional image information of the object and the captured image aligned with the three-dimensional image information; and a terminal device that can communicate with the first server and the second server and displays a display screen including the text data received from the first server and the three-dimensional image information received from the second server, wherein the second server in the information processing system includes: The terminal device identifies the captured image based on the field of view of the three-dimensional image information selected by the terminal device, retrieves the text data acquired when the captured image was taken from the first server, and performs a process to associate the text data with the three-dimensional image information of the field of view, or to associate generated information generated based on the text data with the three-dimensional image information of the field of view. The terminal device performs a process to display the associated text data and three-dimensional image information, or the associated generated information and three-dimensional image information, received from the second server, on the display screen. A program to execute.

Images