Information processing systems, 3D image management servers, information processing methods, programs

Abstract

To generate text information using an appropriate model. [Solution] In the present invention, the three-dimensional image management server 40 includes a first model 4004A that has learned the correspondence between three-dimensional image information of the object and a predetermined region image in a captured image obtained by photographing the object with a photography device, and input information input to the terminal device; a second model 4004B that has learned the correspondence between three-dimensional image information of the object and input information input to the terminal device; and a text information generation unit 45 that generates text information about the object using the three-dimensional image information of the object, the predetermined region image, and the first model selected by the terminal device, or generates text information about the object using the three-dimensional image information of the object selected by the terminal device and the second model.

Description

Technical Field

[0001] The present invention relates to an information processing system, a three-dimensional image management server, an information processing method, and a program.

Background Art

[0002] There is known a generative AI that can generate text information from various contents (articles, images, voices, etc.). Conventional AI only presented the most appropriate answer from the learned data, but generative AI can continuously learn by itself, learn information and data not given by humans, and output original content that has never been input.

[0003] Techniques for improving the quality of learning data in machine learning are known (see, for example, Patent Document 1). Patent Document 1 discloses a technique in which a learning model outputs a failure recovery procedure using the failure information received from a user as an input, and sets an evaluation weight regarding the usefulness of the failure recovery procedure used for re-learning by the learning model based on skill information indicating the user's failure recovery skill.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention provides a technique for generating text information with an appropriate model.

Means for Solving the Problems

[0005] In view of the above problems, the present invention is an information processing system including a three-dimensional image management server that manages three-dimensional image information of an object, and a terminal device capable of communicating with the image management server, The three-dimensional image management server is The system comprises: a first model that has learned the correspondence between the three-dimensional image information of the object, a predetermined region image in the captured image obtained by photographing the object with a photography device, and input information input to the terminal device; a second model that has learned the correspondence between the three-dimensional image information of the object and input information input to the terminal device; and a text information generation unit that generates text information about the object using the three-dimensional image information of the object, the predetermined region image, and the first model selected by the terminal device, or generates text information about the object using the three-dimensional image information of the object selected by the terminal device and the second model. The terminal device has a display control unit that displays a display screen containing the text information. [Effects of the Invention]

[0006] Text information can be generated using an appropriate model. [Brief explanation of the drawing]

[0007] [Figure 1] This is an overall configuration diagram of an example of an information processing system. [Figure 2] This is a hardware configuration diagram of an example of a three-dimensional image management server, a captured image management server, or a terminal device. [Figure 3] This is an example functional configuration diagram illustrating the functions of a 3D image management server, a captured image management server, and terminal devices in an information processing system. [Figure 4] This is a conceptual diagram showing an example of a three-dimensional image information management table according to this embodiment. [Figure 5] This is a conceptual diagram showing an example of a captured image information management table according to this embodiment. [Figure 6] This is a sequence diagram illustrating an example of communication processing between wide-field images and audio data. [Figure 7] This is an explanatory diagram displayed by the terminal device during text information generation processing and model update processing. [Figure 8]This is an explanatory diagram displayed by the terminal device during text information generation processing and model update processing. [Figure 9] This is a sequence diagram showing an example of a model update process. [Figure 10] This is an example flowchart illustrating how the decision-making unit determines whether to update the first tacit knowledge model or the second tacit knowledge model. [Figure 11] This is a diagram showing an example of a property selection screen. [Figure 12] This diagram shows an example of a property management screen. [Figure 13] This figure shows an example of a three-dimensional image display screen. [Figure 14] This figure shows an example of a message that popped up on a 3D image display screen. [Figure 15] This is a sequence diagram showing an example of text information generation processing using the first implicit knowledge model. [Figure 16] This figure shows an example of a property display screen during the inference phase. [Figure 17] This figure shows an example of a message that pops up on the property listing screen. [Figure 18] This figure shows an example of text information displayed on a text display screen. [Figure 19] This sequence diagram illustrates an example of a process where a 3D image management server communicates with an image capture management server to update the model. [Figure 20] This is a sequence diagram showing an example of a model update process. [Figure 21] This is a sequence diagram showing an example of text information generation processing using the second implicit knowledge model. [Figure 22] This figure shows an example text display screen containing text information generated based on the second tacit knowledge model. [Figure 23] This is a sequence diagram showing an example of a model update process. [Figure 24] This is an example flowchart illustrating how the decision-making unit determines whether to update the first tacit knowledge model or the second tacit knowledge model. [Figure 25] This is a diagram showing a property display screen of an example. [Figure 26] This is a sequence diagram showing an example of text information generation processing in which a terminal device directly logs in to a three-dimensional image management server and uses a second tacit knowledge model. [Figure 27] This is a diagram showing a property display screen of an example when a terminal device directly logs in to a three-dimensional image management server. [Figure 28] This is a diagram showing a text display screen of an example. [Figure 29] This is an overall configuration diagram of an example of an information processing system. [Figure 30] This is a functional configuration diagram of an example explaining the functions of a three-dimensional image management server, a photographed image management server, and a terminal device in an information processing system. [Figure 31] This is a sequence diagram of an example showing generation processing of text information and image information. [Figure 32] This is a diagram showing the generated image information displayed on a captured image display screen.

Mode for Carrying Out the Invention

[0008] Hereinafter, as an example of a mode for carrying out the present invention, an information processing system and an information processing method performed by the information processing system will be described with reference to the drawings.

[0009] <Supplementary Explanation Regarding Tacit Knowledge> In industries such as civil engineering and architecture, BIM / CIM is being promoted for the purpose of addressing issues such as the declining birthrate and aging population, and improving labor productivity.

[0010] BIM is an abbreviation for Building Information Modeling, and it is a solution for utilizing information in all processes from building design, construction to maintenance management using a database of a building that adds attribute data such as cost, finish, and management information to a three-dimensional digital model of a building (hereinafter referred to as a 3D model) created on a computer.

[0011] CIM stands for Construction Information Modeling and is a solution for the civil engineering sector (including infrastructure such as roads, power, gas, and water) that was proposed following the example of BIM, which was being developed in the building sector. Similar to BIM, CIM aims to improve the efficiency and sophistication of the entire construction production system by sharing information among stakeholders, primarily using 3D models.

[0012] A crucial aspect of promoting BIM and CIM implementation is how to effectively utilize the BIM and CIM systems that have been constructed.

[0013] Specifically, 3D models restored using BIM and CIM can be used not only for design and construction purposes, but also for other tasks such as maintenance and site surveys. In other words, they can be used not only as design drawings, but also for other purposes such as recording information in the 3D model and sharing it with others.

[0014] Furthermore, since the work performed on the 3D model can be recorded as a log, if tacit knowledge can be extracted from this data, it can be effectively used for transferring skills from experienced personnel to younger ones. This is expected to lead to front-loading of operations and talent development.

[0015] Focusing on the transmission of tacit knowledge, the challenge lies not only in 3D models but also in 2D data (such as 3D spherical images and planar images), as described above, in how to transmit tacit knowledge between different tasks and among users with varying levels of skill.

[0016] Specifically, because tacit knowledge is qualitative and difficult to quantify, even if an tacit knowledge model is generated from tacit knowledge, it is difficult to secure user trust in the tacit knowledge model, making it difficult to promote its use. For example, if the field of expertise of the tacit knowledge model differs from that of the user, no matter how excellent the tacit knowledge model is, it will have no value to the user. Similarly, if the knowledge level of the tacit knowledge model is lower than the user's knowledge level, it will have no value to the user.

[0017] However, it is also true that tacit knowledge models can give users new perspectives and insights, and by using tacit knowledge models, even inexperienced users have great potential to acquire know-how and skills and apply them to their work.

[0018] Furthermore, the first and second servers may each manage related information. For example, the first server might hold property management information (such as captured images and audio text of the property), while the second server manages the property's 3D image information. Assuming the second server creates a tacit knowledge model, it can generate a first tacit knowledge model (an example of the first model) using captured images and audio text of the item, or a second tacit knowledge model (an example of the second model) that does not use captured images and audio text of the item. In this case, for example, the first tacit knowledge model can be used when you want to generate expert knowledge and text information specific to a particular item, while the second tacit knowledge model can be used when you want to generate expert knowledge and general text information applicable to similar items in general. If the user can specify which tacit knowledge model to use, they can choose between tacit knowledge specific to a particular item or tacit knowledge applicable to similar items in general.

[0019] Therefore, in this embodiment, we will describe a system in which the user can switch between a first tacit knowledge model that uses property management information (such as captured images and audio text of the property) and a second tacit knowledge model that does not use property management information. Property management information is, for example, captured images and audio text of items, but is not limited to these.

[0020] <About Terminology> A user is someone who uses the text information (or other content such as images) generated by the tacit knowledge model. A data provider is someone who provides the data (such as audio information, text information, operation information, images, 3D data, etc.) that the tacit knowledge model uses for training.

[0021] Tacit knowledge is knowledge based on an individual's experience, intuition, etc. An tacit knowledge model is a model that learns tacit knowledge and outputs answers based on that learned tacit knowledge in response to questions. A model is a mechanism or artificial intelligence (AI) that learns the correspondence between input data and output data and outputs output data for given input data. Note that the output data does not depend on whether or not there is training data.

[0022] A property can be any space where items can be placed, such as a facility or a room within a facility. Items are things that are placed in the property. The types of items placed will vary depending on the function of the facility.

[0023] For example, properties include real estate, factories, construction sites, research facilities, medical facilities, farmland, warehouses, and equipment requiring maintenance. Goods include furniture, materials, equipment, heavy machinery, tools, instruments, materials, cultured products, and food.

[0024] The object is an object that is photographed by a photographic device, and whose state can be managed primarily by recording it as an image. In this embodiment, it will be described using the term "article." The object is, for example, placed on a property.

[0025] Three-dimensional image information of an object is an image of a 3D model captured by a virtual camera. The user can change the viewpoint of the three-dimensional image information.

[0026] Generated information is information generated based on three-dimensional image information and captured images. Generated information may be generated by an implicit knowledge model. In this embodiment, generated information is described using the terms implicit knowledge comments or text information.

[0027] A display screen is a screen in which one or more pieces of information, such as three-dimensional image information, captured images, or generated information, are displayed on a single screen.

[0028] A wide-field image is an image with a wide field of view that includes areas that cannot be seen with a normal field of view. For example, a wide-field image is an image with a wide field of view that is captured with a wide shooting range, including a 360-degree image (also called a spherical image, omnidirectional image, or all-around image) that captures the entire 360 ​​degrees.

[0029] A predetermined region image is an image corresponding to a predetermined region within a wide-field image. The predetermined region image is projected onto a two-dimensional plane and is therefore a planar image. In this embodiment, the predetermined region image is saved by a capture operation and is therefore called a captured image.

[0030] [First Embodiment] <Example System Configuration> Figure 1 is an overall configuration diagram of an information processing system 100 according to an embodiment of the present invention. The information processing system 100 of this embodiment includes a terminal device 10, which is an example of an input / output device, a shooting device 5, a three-dimensional image management server 40, and a captured image management server 20. However, the terminal device 10 may be connected to the three-dimensional image management server 40 or the captured image management server 20 as needed, and the information processing system 100 does not necessarily have to include it.

[0031] The three-dimensional image management server 40 (an example of a second server) is one or more information processing devices that can communicate with the terminal device 10 via a communication network N. The three-dimensional image management server 40 manages three-dimensional image information of objects and also has an implicit knowledge model and a large-scale language model. The three-dimensional image management server 40 uses these to return text information containing implicit knowledge to the user. The three-dimensional image management server 40 may be a web server that returns processing results to the terminal device 10 in response to requests from the terminal device 10. A server is a computer or software that performs the function of providing information and processing results in response to requests from a client.

[0032] The 3D image management server 40 may support cloud computing. Cloud computing refers to a usage model in which network resources are utilized without being aware of specific hardware resources. Cloud computing can take the form of SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service), and any of these forms is acceptable. Therefore, the 3D image management server 40 does not need to be housed in a single enclosure or provided as a single unit of equipment. The functions of the 3D image management server 40 may be distributed among multiple information processing devices, or multiple information processing devices may each have all the functions, and the information processing device may be switched by load balancing or other means.

[0033] Furthermore, instead of the three-dimensional image management server 40 having an implicit knowledge model and a large-scale language model, the three-dimensional image management server 40 may call an API (Application Programming Interface) exposed by an external system and utilize at least one of the implicit knowledge model or the large-scale language model.

[0034] The image capture management server 20 (an example of the first server) is one or more information processing devices that can communicate with the terminal device 10 via the communication network N. The image capture management server 20 manages property management information. Property management information is, for example, a string of characters such as text. The image capture management server 20 does not have three-dimensional image information. The image capture management server 20 can distribute live images of wide-field images captured by the camera 5. The image capture management server 20 also manages captured images captured by users. The image capture management server 20 is a server that allows users to manage the progress of construction work on a property and the placement of items while viewing video data of the property.

[0035] The captured image management server 20 may be a web server that returns processing results to the terminal device 10 in response to requests from the terminal device 10. The captured image management server 20 can communicate with the three-dimensional image management server 40 via the communication network N. The captured image management server 20 may also support either cloud computing or on-premises deployment.

[0036] It is preferable that the 3D image management server 40 and the captured image management server 20 are linked to a degree that enables single sign-on. The 3D image management server 40 can communicate with the captured image management server 20 via an API exposed by the captured image management server 20. Alternatively, the 3D image management server 40 and the captured image management server 20 may have a business partnership.

[0037] Terminal device 10 is a general-purpose information processing terminal used by users of the information processing system 100. Terminal device 10 runs a web browser and a dedicated native application for the 3D image management server 40 or the captured image management server 20. When terminal device 10 runs a web browser, terminal device 10 and the 3D image management server 40 or the captured image management server 20 run a web application. A web application is an application that operates through the cooperation of a program written in a programming language (e.g., JavaScript®) that runs on a web browser and a program on the web server (3D image management server 40). When a web application is executed, the processing of this embodiment may be performed by the 3D image management server 40 or the captured image management server 20, or by terminal device 10 that receives the web application.

[0038] An application that cannot be executed unless it is installed on the terminal device 10 is called a native application. In this embodiment as well, the application executed on the terminal device 10 may be either a web application or a native application. In this case as well, the processing in this embodiment may be performed by the three-dimensional image management server 40 or by the terminal device 10 that is executing the native application.

[0039] Terminal device 10 can be, for example, a PC (Personal Computer), a smartphone, a PDA (Personal Digital Assistant), or a tablet device. Alternatively, terminal device 10 can be any device capable of running a web browser or native applications. Terminal device 10 may also be an electronic whiteboard, a television receiver, a glasses device, or a wearable device. Furthermore, there may be multiple terminal devices 10.

[0040] The terminal device 10 can communicate with the three-dimensional image management server 40 and the captured image management server 20 via the communication network N. The communication network N is constructed using the Internet, LAN (Local Area Network), provider services, etc. The communication network N may include not only wired communication but also mobile communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), and wireless LAN networks. Furthermore, the terminal device 10 can communicate using short-range communication technologies such as Bluetooth (registered trademark) and NFC (registered trademark, Near Field Communication).

[0041] The imaging device 5 is a digital camera for obtaining wide-field images and recording sound. The imaging device 5 is connected to the communication network N via the relay device 3. The relay device 3 has the function of a cradle for charging the imaging device 5 and transmitting and receiving data. In addition, the relay device 3 can communicate data with the imaging device 5 via contacts and can also communicate data with the captured image management server 20 via the communication network N. The imaging device 5 and the relay device 3 are installed at predetermined locations on site Sa, such as construction sites, exhibition halls, educational institutions, and medical facilities. The imaging device 5 may be a general digital camera that obtains narrow-field images, such as a single-lens reflex camera, and the captured image management server 20 may distribute live images of the narrow-field images captured by the imaging device 5. In this case, the predetermined area image is an image corresponding to all or part of a predetermined area of ​​the captured image.

[0042] In Figure 1, the 3D image management server 40, the captured image management server 20, and the terminal device 10 communicate via the communication network N. However, the user may operate the 3D image management server 40 or the captured image management server 20 directly from a console, or the terminal device 10 may have the functions of the 3D image management server 40 or the captured image management server 20. In other words, the terminal device 10 may provide the functions of the information processing system 100 in a standalone form.

[0043] <Example Hardware Configuration> Figure 2 is a hardware configuration diagram of the three-dimensional image management server 40, the captured image management server 20, or the terminal device 10 according to this embodiment. Each hardware component of the three-dimensional image management server 40 or the captured image management server 20 is indicated by a 400-series reference numeral. Each hardware component of the terminal device 10 is indicated by a 100-series reference numeral.

[0044] The hardware configuration of the terminal device 10 will be described below, but the hardware configuration of the three-dimensional image management server 40 or the captured image management server 20 is the same and will therefore not be described.

[0045] The terminal device 10 is built using a computer and, as shown in Figure 2, includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HD (Hard Disk) 104, HDD (Hard Disk Drive) controller 105, display I / F 106, and communication I / F 107.

[0046] Of these components, the CPU 101 controls the overall operation of the terminal device 10. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101.

[0047] HD104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to HD10 according to the control of the CPU 101.

[0048] The display I / F 106 is a circuit that displays images on the display 106a. The display 106a is a type of display unit such as a liquid crystal or organic EL (electroluminescence) that displays various information such as cursors, menus, windows, characters, or images. The communication I / F 107 is an interface used for communication with other devices.

[0049] If the terminal device 10 is a glass device, the terminal device 10 may use a circuit that displays an image on a lens or the like, which is a transmissive reflective material, instead of the display I / F 106.

[0050] Communication I / F107 is, for example, a NIC (Network Interface Card) that supports TCP (Transmission Control Protocol) / IP (Internet Protocol).

[0051] Furthermore, the terminal device 10 is equipped with a sensor I / F 108, an audio input / output I / F 109, an input I / F 110, a media I / F 111, and a DVD-RW (Digital Versatile Disk Rewritable) drive 112.

[0052] The sensor interface 108 is an interface for receiving detection information from various sensors. The sound input / output interface 109 is a circuit that processes the input and output of sound signals between the speaker 109a and the microphone 109b according to the control of the CPU 101. The input interface 110 is an interface for connecting a predetermined input means to the terminal device 10.

[0053] Keyboard 110a is a type of input device equipped with multiple keys for inputting characters, numbers, various instructions, etc. Mouse 110b is a type of input device used for selecting and executing various instructions, selecting processing targets, moving the cursor, and operating on the display screen, etc.

[0054] The media interface 111 controls the reading or writing (storage) of data to or from a recording medium 111a such as flash memory. The DVD-RW drive 112 controls the reading or writing of various types of data to or from a DVD-RW 112a, which is an example of a removable recording medium. Note that the DVD-RW may be a DVD-R or the like. The DVD-RW drive 112 may also be a Blu-ray drive that controls the reading or writing of various types of data to or from a Blu-ray Disc (registered trademark).

[0055] Furthermore, the terminal device 10 is equipped with a bus line 113. The bus line 113 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 101.

[0056] Furthermore, recording media such as HDs and CD-ROMs on which the above programs are stored can be provided domestically or internationally as program products. The terminal device 10 realizes the information processing method according to the present invention by executing, for example, the program according to the present invention.

[0057] <About the features> Figure 3 is an example of a functional configuration diagram illustrating the functions of the three-dimensional image management server 40, the captured image management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment. The capturing device 5 and the relay device 3 are assumed to have existing functions.

[0058] <<Terminal device>> As shown in Figure 3, the terminal device 10 includes a transmitting / receiving unit 11, an input receiving unit 12, a display control unit 13, an audio control unit 14, a conversion unit 15, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 101 that follow a program deployed from the HD 104 onto the RAM 103. The terminal device 10 also has a storage unit 1000 constructed from at least one of the RAM 103 and HD 104 shown in Figure 2.

[0059] The transmitting / receiving unit 11 is an example of a transmission means and is implemented by commands from the CPU 101 shown in Figure 2 and the communication I / F 107, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0060] The input receiving unit 12 is an example of an input receiving means and is mainly implemented by commands from the CPU 101 shown in Figure 2, as well as the input I / F 110 and the sound input / output I / F 109, and accepts various inputs from the user using the microphone 109b, keyboard 110a and mouse 110b.

[0061] The display control unit 13 is an example of a display control means and output means, and is implemented by commands from the CPU 101 shown in Figure 2 and the display I / F 106, and causes various images and screens to be displayed on the display 106a, which is an example of a display unit. If the terminal device 10 is a glasses device, the display control unit 13 causes a virtual image to be displayed on a transmissive reflective member such as a lens instead of the display I / F 106.

[0062] The audio control unit 14 is an example of an audio control means and output means, and is implemented by commands from the CPU 101 shown in Figure 2 and the audio input / output I / F 109, causing the speaker 109a, which is an example of an audio playback unit, to play sound.

[0063] The conversion unit 15 is an example of a processing means, and is implemented by instructions from the CPU 101 shown in Figure 2. It performs processing to convert text information into audio information and processing to convert audio information into text information.

[0064] The storage / reading unit 19 is an example of a storage control means and is executed by instructions from the CPU 101 shown in Figure 2, as well as by the HD 104, media I / F 111, and DVD-RW drive 112. It performs processing such as storing various data in the storage unit 1000, recording media 111a, and DVD-RW 112a, and reading various data from the storage unit 1000, recording media 111a, and DVD-RW 112a.

[0065] <Functional Configuration of the 3D Image Management Server> The three-dimensional image management server 40 includes a transmitting / receiving unit 41, a screen generation unit 42, a judgment unit 43, a specification unit, a text information generation unit 45, an update unit 46, a processing unit 47, a determination unit 48, and a storage / reading unit 49. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 following a program deployed from the HD 404 onto the RAM 403. The three-dimensional image management server 40 also has a storage unit 4000 constructed from the HD 404 shown in Figure 2. The storage unit 4000 is an example of a storage means.

[0066] In Figure 3, all functions are installed in a single 3D image management server 40, but the 3D image management server 40 may also be configured to distribute each function across multiple computers.

[0067] The transmitting / receiving unit 41 is an example of a transmission means and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0068] The screen generation unit 42 is an example of a screen generation means and is implemented by instructions from the CPU 401 shown in Figure 2, generating various screens. When the terminal device 10 runs a web application, screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc. Therefore, screen information may also be referred to as a web application. When the terminal device 10 runs a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0069] The decision unit 43 is an example of a decision-making mechanism, and is implemented by instructions from the CPU 401 shown in Figure 2, and performs various decisions described later.

[0070] The identification unit 44 is an example of an identification means, and is implemented by instructions from the CPU 401 shown in Figure 2, and identifies the target image.

[0071] The text information generation unit 45 is an example of a text information generation means, and is implemented by instructions from the CPU 401 shown in Figure 2. It obtains implicit knowledge comments from an implicit knowledge model and generates text information based on a large-scale language model 4005.

[0072] The update unit 46 is an example of an update means, and is implemented by instructions from the CPU 401 shown in Figure 2, and performs an update of the implicit knowledge model, which will be described later.

[0073] The processing unit 47 is implemented by instructions from the CPU 401 shown in Figure 2 and performs the process of associating three-dimensional image information with generated information (text information generated based on three-dimensional image information and captured images). Alternatively, the processing unit 47 performs the process of associating three-dimensional image information and captured images with generated information. This process includes the process of displaying both on a single screen, or the process of obtaining an implicit knowledge comment, which is an example of text information, from the implicit knowledge model 4004 using the captured image and three-dimensional image information. The processing unit 47 requests processing from the screen generation unit 42 or the text information generation unit 45, etc., depending on the content of the processing.

[0074] The decision unit 48 is implemented by instructions from the CPU 401 shown in Figure 2, and determines whether to use the first implicit knowledge model 4004A or the second implicit knowledge model 4004B. The following are examples of the decision methods. • User choice. When certain conditions are met, the decision unit 48 makes a decision automatically (without confirming with the user) or semi-automatically (by recommending to the user and requesting confirmation). Specific conditions include, for example, cases where the question in the input information (voice, text) relates to an item (e.g., to the condition of an item). In this case, the second tacit knowledge model 4004B may not be able to generate appropriate text information, so it is preferable to select the first tacit knowledge model 4004A.

[0075] The storage / reading unit 49 is an example of a storage control means and is executed by instructions from the CPU 401 shown in Figure 2, as well as by the HD 404, media I / F 411, and DVD-RW drive 412. It performs processing such as storing various data in the storage unit 4000, recording media 411a, and DVD-RW 412a, and reading various data from the storage unit 4000, recording media 411a, and DVD-RW 412a. The storage unit 4000, recording media 411a, and DVD-RW 412a are examples of storage means.

[0076] The memory unit 4000 contains a three-dimensional image information management DB 4001, a model shape management DB 4002, a caption model 4003, a first tacit knowledge model 4004A, a second tacit knowledge model 4004B, and a large-scale language model 4005.

[0077] The 3D image information management DB 4001 manages 3D image information of items placed on the property. This 3D image information is information that visually represents the items (also called models) placed on the property. The model shape management DB 4002 manages 3D model shape information of items placed on the property. The 3D image management server 40 can generate 3D image information related to the property from the 3D model shape information. The 3D model shape information is information that renders the item in 3D, such as a 3D model of the item or a 3D point cloud. The 3D model shape information may be composed of polygons or CAD. In addition, the 3D image information management DB 4001 or the model shape management DB 4002 may store wide-field images such as 360-degree images of the property.

[0078] Caption model 4003 is a model that is generated by performing a training process using image and caption comment combinations as training data, and enables a computer to function to output caption comments based on images. Caption comments are explicit knowledge and are used as a term in contrast to tacit knowledge. Here, caption comments are text data, such as comments expressed in audio or text that describe an image. Caption comments relating to objects or articles are associated with the identification information of those objects or articles.

[0079] The first tacit knowledge model 4004A is generated by performing a learning process using the correspondence between three-dimensional image information and captured images and the tacit knowledge (input information, speech text, etc.) associated with this three-dimensional image information and captured images as training data, and is a model that makes the computer function to output tacit knowledge comments based on images. The tacit knowledge model 4004 learns by associating information as follows. Correspondence between three-dimensional image information and captured images and input information • Correspondence between three-dimensional image information and captured images and audio text. Correspondence between three-dimensional image information and captured images, and audio text and input information. Here, tacit knowledge comments are text data, and are comments expressed in audio or text, excluding caption comments, that is, comments relating to content not represented in the image.

[0080] The second tacit knowledge model 4004B does not use captured images for training. In other words, the second tacit knowledge model 4004B is generated by performing a training process using the correspondence between three-dimensional image information of an item and input information as training data, and it is a model that makes the computer function to output tacit knowledge comments based on the three-dimensional image information.

[0081] The large-scale language model 4005 is a computer language model that is generated by performing a training process using a vast amount of unlabeled text as training data and consists of an artificial neural network with many parameters. The large-scale language model 4005 can capture much of the syntax and meaning of human language by being sufficiently trained with context-learning techniques such as next sentence prediction, which understands context by determining whether sentence 1 and sentence 2 are consecutive, and the masked language model, which understands context by masking words in a sentence and predicting the masked words from the words before and after them.

[0082] <<Three-dimensional image information management table>> Figure 4 is a conceptual diagram showing an example of a three-dimensional image information management table according to this embodiment. The storage unit 4000 has a three-dimensional image information management DB 4001 constructed, which is composed of the three-dimensional image information management table shown in Figure 4. In the three-dimensional image information management table shown in Figure 4, the model ID, location information, and captured image are associated and managed in relation to the property identification information.

[0083] Property identification information is an example of property identification information used to identify a property. A property can be any space where goods can be placed, such as a facility or a room within a facility. The types of goods placed will vary depending on the function of the facility. A property can be represented in an easily manageable unit, such as "○○ Building 2F-N (the north side of the 2nd floor of ○○ Building)."

[0084] A Model ID is an example of identification information used to identify items placed in a property. The items may consist of three-dimensional model shape information, such as polygons or CAD data, in the Model Shape Management DB4002. The Model ID links the three-dimensional image information to the three-dimensional model shape in the Model Shape Management DB4002.

[0085] Location information is information that indicates the position of an item's model in a three-dimensional virtual space, where the item is represented in a virtual space, using three-dimensional coordinates (XYZ). For example, location information is indicated by the three-dimensional coordinates of eight points that define the rectangular space occupied by the model.

[0086] This location information is determined as the location information (latitude, longitude, and altitude) of the relay device 3 by GNSS (Global Navigation Satellite System) satellites such as GPS (Global Positioning Systems), or by IMES (Indoor Messaging System) as an indoor GPS. Indoor positioning methods include Wi-Fi positioning, RFID (Radio Frequency Identifier) ​​positioning, beacon positioning, pedestrian autonomous navigation positioning, geomagnetic positioning, acoustic positioning, and UWB (Ultra Wide Band) positioning.

[0087] The captured image is a captured image (two-dimensional image) obtained from the captured image management server 20. Capture means that a still image of a specific moment has been captured. The captured image is an image in which a predetermined area, specified by the field of view, has been cropped from the wide-field image captured by the shooting device 5. The reason why the captured image is registered in association with an item is that the three-dimensional image information is composed of a 3D model. When a user clicks on an item in the three-dimensional image information, the item (model ID) is identified by its coordinates. Alternatively, the three-dimensional image management server 40 may determine the position and field of view of the virtual camera based on the position information and field of view information of the shooting device 5 in the live image, and identify the model of the item that falls within the field of view from this position. The three-dimensional image management server 40 associates the captured image with the model ID of the identified item. Therefore, the captured image may contain an item.

[0088] The location information in Figure 4 is managed in correspondence with absolute positions on Earth. For example, by associating the origin of the location information in Figure 4 (X=0, Y=0, Z=0) with absolute positions on Earth (latitude, longitude, altitude), all coordinates in three-dimensional images, including three-dimensional models and objects, are associated with absolute positions on Earth.

[0089] In addition to these, instruction manuals, daily reports, quotations, drawings, etc., may also be registered in the three-dimensional image information management table.

[0090] <Functional configuration of the image management server> Let's return to Figure 3 for explanation. The captured image management server 20 has a transmitting / receiving unit 21, a screen generation unit 22, and a storage / reading unit 29. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 following a program deployed from the HD 404 onto the RAM 403. The captured image management server 20 also has a storage unit 2000 constructed from the HD 404 shown in Figure 2. The storage unit 2000 is an example of a storage means.

[0091] In Figure 3, all functions are installed in a single image management server 20, but the image management server 20 may also be configured to distribute each function across multiple computers.

[0092] The transmitting / receiving unit 21 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0093] The screen generation unit 22 is an example of a screen generation means and is implemented by instructions from the CPU 401 shown in Figure 2, generating various screens. When the terminal device 10 runs a web application, screen information is created using HTML, XML, CSS, and JavaScript (registered trademark), etc. Therefore, screen information may also be referred to as a web application. When the terminal device 10 runs a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0094] The storage / reading unit 29 is an example of a storage control means and is executed by instructions from the CPU 401 shown in Figure 2, as well as by the HD 404, media I / F 411, and DVD-RW drive 412. It performs processing such as storing various data in the storage unit 2000, recording media 411a, and DVD-RW 412a, and reading various data from the storage unit 2000, recording media 411a, and DVD-RW 412a. The storage unit 2000, recording media 411a, and DVD-RW 412a are examples of storage means.

[0095] <<Image Information Management Table>> Figure 5 is a conceptual diagram showing an example of an image capture information management table according to this embodiment. The memory unit 2000 has an image capture information management DB 2001 constructed, which is composed of an image capture information management table as shown in Figure 5.

[0096] In this image information management table, live images and captured images are recorded in association with property identification information. In the image information management table, the date and time of "shooting and sound collection", shooting location, field of view information, current date and time audio text (shooting device), and current date and time audio text (communication terminal) are stored and managed in association with the property identification information. The position of shooting device 5 was determined by the relay device 3 to which shooting device 5 is attached. Alternatively, shooting device 5 may determine its own position. • The "Capture and Sound Collection" date and time indicates the date and time of the live image captured and sound collected by the capture device 5. In the diagram, the live image is captured every second, but it may also be captured at 30fps or other rates. The shooting position indicates the position of the shooting device 5 (absolute position on Earth) when the wide-field image is captured. As will be described later, a terminal used to view live images in a meeting, etc., is called a communication terminal, and the operation in which a user of the communication terminal saves a captured image of the field of view they wish to save from the wide-field image is called a capture image. The image saved by the capture operation is called a captured image. This captured image is sent to the three-dimensional image management server 40. The shooting position is also the sound collection position. The field of view information is information used to identify a predetermined area displayed by the communication terminal when the user performs the capture operation from the wide-field image. The audio text registered in the "Current Time Audio Text (Capture Device)" field is text data converted by speech recognition from audio collected by the capture device 5. The audio text is comment data about items spoken by meeting participants while viewing the live image. The audio text registered in the "Current Time Audio Text (Communication Terminal)" field is text data converted by speech recognition from the speech of participants viewing live images on their communication terminals. The audio text is comment data about items spoken by meeting participants while viewing live images.

[0097] <Sending content data> Figure 6 is a sequence diagram showing the communication processing of wide-field images and audio data. In this embodiment, we will describe the case where the imaging device 5, participant A's communication terminal 9a, and participant B's communication terminal 9b are participating in the same remote communication. Steps S201 to S204 in Figure 6 are repeated.

[0098] S201: The camera 5 captures images of the surroundings and collects sound, sending video data (wide-field image) and audio data to the relay device 3. The camera 5 also transmits a device ID to identify the camera 5 in order to identify the property. As a result, the relay device 3 acquires the video data and audio data. The captured image management server 20 has pre-associated device IDs with properties.

[0099] S202: The relay device 3 transmits the acquired video data, audio data, and device ID to the captured image management server 20 via the communication network N. As a result, the transmitting / receiving unit 21 of the captured image management server 20 receives the video data, audio data, and device ID. The captured image management server 20 identifies the object by the device ID. As a result, the live image and the date and time of capture and sound collection are saved, for example, every second by the storage / reading unit 29 in the captured image information management DB 2001. Note that the live image may not be stored and only distributed. The captured image management server 20 (or an existing speech recognition server) uses the audio data to convert the audio portion into text data (hereinafter referred to as "speech text"). The storage / reading unit 29 saves the speech text in the captured image information management DB 2001.

[0100] S203a: The captured image management server 20 reads the participant IDs of participants in the same conference as the camera 5 from the conference information. The captured image management server 20 also reads the IP addresses of communication terminals 9a and 9b based on the read participant IDs. The captured image management server 20 refers to the IP address of communication terminal 9a and transmits the received video data and audio data to communication terminal 9a. As a result, communication terminal 9a receives the video data and audio data, displays the wide-field image, and outputs sound.

[0101] S203b: Similarly, the captured image management server 20 refers to the IP address of the communication terminal 9b and sends video data and audio data to the communication terminal 9b. As a result, the communication terminal 9b displays the wide-field image and outputs sound.

[0102] S204a, 204b: Meanwhile, communication terminals 9a and 9b transmit audio data from participants A and B to the captured image management server 20. This audio data is generated when microphones capture the voices of participants A and B operating the communication terminals 9a and 9b and convert them into audio data. The storage and reading unit 29 of the captured image management server 20 stores the audio text in the captured image information management DB 2001.

[0103] S205: Users of communication terminals 9a and 9b can change the viewpoint of the video data, which is a wide-field image. If participant B wants to save a predetermined area of ​​the wide-field image displayed after changing the viewpoint, participant B can perform a capture operation at any time. Upon receiving the capture operation, communication terminal 9b sends a capture request and field of view information indicating the predetermined area currently displayed on the display to the captured image management server 20.

[0104] S206: When the captured image management server 20 receives a capture request and field of view information, it identifies the IP address of the relay device 3 that is participating in the same conference as the communication terminal 9b, and sends the capture request and field of view information to it.

[0105] S207: Relay device 3 receives the capture request and field of view information and transmits it to the shooting device 5.

[0106] S208: When the imaging device 5 receives a capture request, it generates a captured image based on the field of view information. The imaging device 5 transmits the captured image, shooting position, and field of view information to the relay device 3.

[0107] S209: The relay device 3 transmits the captured image, shooting location, and field of view information to the captured image management server 20. The captured image management server 20 identifies the property by device ID, similar to step S203. The storage / reading unit 29 stores the captured image, shooting location, and field of view information in the captured image information management DB 2001.

[0108] Through the above process, the captured image information management DB2001 stores wide-field images (live images) and audio data in real time. If a participant performs a capture operation, the captured image, shooting location, and field of view image are also saved.

[0109] In Figure 6, the camera 5 generates a captured image in response to a request from the communication terminal 9b. However, the communication terminal 9b may generate a captured image of a predetermined area that it is currently displaying and send it to the captured image management server 20.

[0110] <An example of model updating and text information generation> Refer to Figures 7 and 8 to explain how to update the model and generate text information. In Figures 7 and 8, speech-to-text is not used for updating the model and generating text information, but the learning process can be similarly achieved by replacing or adding dialogue Q1, etc., with speech-to-text.

[0111] Figure 7 is an explanatory diagram of what the terminal device 10 displays during text information generation processing and model update processing. Figure 7(a) is an explanatory diagram of the model update processing. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the three-dimensional image management server 40 on the display 106a, and the display screen 900 includes the target image 1100 and text 1200.

[0112] The input receiving unit 12 of the terminal device 10 receives audio information from the microphone 109b, which represents the dialogue Q1, A1, Q2, and A2 between data providers M1 and M2, as input information to be entered by the data provider into the displayed screen 900. It is preferable that data providers M1 and M2 have extensive knowledge, including tacit knowledge, about their work. Through such a dialogue between data providers M1 and M2, the tacit knowledge model is updated, allowing the user to obtain useful tacit knowledge comments.

[0113] The identification unit 44 identifies the target image 1100, which is the portion of the display screen 900 excluding the text 1200.

[0114] The judgment unit 43 then uses the target image 1100 to determine the degree of relevance between the caption comments obtained from the caption model 4003 and the dialogues Q1, A1, Q2, and A2.

[0115] The update unit 46 updates the tacit knowledge model by using comments from the dialogues Q1, A1, Q2, and A2 that are judged to have a low degree of relevance as tacit knowledge comments, along with the target image 1100, etc., as training data, and updates the caption model 4003 by using comments that are judged to have a high degree of relevance as caption comments, along with the target image 1100, as training data.

[0116] Therefore, the tacit knowledge model learns the correspondence between the target image 1100 and the dialogues Q1, A1, Q2, and A2. The target image 1100 has features extracted using several feature extraction models suitable for images, such as a CNN (Convolutional Neural Network). These features represent, for example, what is depicted and where, or the tasks being performed within the image. Thus, the tacit knowledge model can learn the correspondence between the image features and the dialogues Q1, A1, Q2, and A2.

[0117] Figure 7(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the three-dimensional image management server 40 on the display 106a, and the display screen 900 includes an image 1110 and text 1210.

[0118] The input receiving unit 12 of the terminal device 10 receives audio information indicating questions Q11 and Q12 from user M3 via the microphone 109b as input information to be entered by the user on the displayed screen 900.

[0119] The identification unit 44 identifies the image 1110 that does not contain the text 1210 as the target image.

[0120] The text information generation unit 45 uses image 1110 to acquire tacit knowledge comments based on the tacit knowledge model. The tacit knowledge model extracts features from image 1110 and determines that the features of image 1110 in Figure 7(b) are similar to those of image 1110 at the time of update, thereby identifying dialogues Q1, A1, Q2, and A2 related to image 1110. Dialogues Q1, A1, Q2, and A2 become tacit knowledge comments.

[0121] Furthermore, the text information generation unit 45 uses tacit knowledge comments (dialogue Q1, A1, Q2, A2) and questions Q11, Q12, etc., to generate text information related to the answers A11 and A12 to questions Q11 and Q12, respectively, based on the large-scale language model 4005.

[0122] The display control unit 13 of the terminal device 10 displays the text information related to the answers A11 and A12 received from the three-dimensional image management server 40 on the display 106a.

[0123] Figure 8 is another explanatory diagram displayed by the terminal device 10 during the model update process and text information generation process according to this embodiment. Figure 8 illustrates a case in which no question text is used during model update and text information generation.

[0124] Figure 8(a) is an explanatory diagram of the model update process. Figure 8(a) shows an example in which the tacit knowledge model is updated with audio information and partial images from a single data provider, rather than a conversation between data providers.

[0125] The display control unit 13 of the terminal device 10 displays the display screen 900 received from the three-dimensional image management server 40 on the display 106a, and the display screen 900 includes a first image 1100A and a second image 1100B.

[0126] The input receiving unit 12 of the terminal device 10 receives character information from the keyboard 110a, which represents comments C1 to C4 from data provider M4, as input information to be entered by the data provider in response to the displayed screen 900.

[0127] Furthermore, the input receiving unit 12 receives operation information from the mouse 110b, which indicates an operation by the data provider M4 to identify a partial image 1100B1 in the second image 1100B, as input information to be entered by the data provider M4 into the displayed screen 900.

[0128] The identification unit 44 may identify the partial image 1100B1 as the target image, or it may identify the first image 1100A or the second image 1100B as the target image.

[0129] The judgment unit 43 then uses the target image to determine the degree of relevance between the caption comments obtained from the caption model 4003 and comments C1 to C4.

[0130] The update unit 46 updates the tacit knowledge model by using comments C1 to C4 that are judged to have low relevance as tacit knowledge comments, along with partial image 1100B1, etc., as training data, and updates the caption model 4003 by using comments that are judged to have high relevance as caption comments, along with partial image 1100B1, as training data.

[0131] Therefore, the tacit knowledge model learns the correspondence between the partial image 1100B1 and the comments C1-C4. Features are extracted from the partial image 1100B1 using several feature extraction models suitable for images, such as CNNs. These features represent, for example, what is depicted and where, or the work being performed. Thus, the tacit knowledge model can learn the correspondence between the image features and the comments C1-C4.

[0132] Figure 8(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the three-dimensional image management server 40 on the display 106a, and the display screen 900 includes the image 1110.

[0133] User M5 does not input anything to the displayed screen 900, and the input reception unit 12 does not accept any input information from the user to the displayed screen 900. The identification unit 44 identifies the entire displayed screen 900, which is image 1110, as the target image.

[0134] When user M5 performs an operation to identify a partial image 1100B1 on the display screen 900, the input receiving unit 12 receives operation information from the mouse 110b indicating the operation to identify the partial image as input information. In this case, the identification unit 44 identifies the partial image on the display screen 900 as the target image according to the operation information.

[0135] The text information generation unit 45 uses the partial image 1100B1 to acquire tacit knowledge comments based on the tacit knowledge model. The tacit knowledge model determines that the features of image 1110B1 in Figure 8(b) are similar to those of image 1110B1 at the time of update, and can identify comments C1 to C4 related to image 1110B1. The tacit knowledge model extracts comments C1 to C4 as tacit knowledge comments. The text information generation unit 45 generates text information related to comments C11 to C14 based on the large-scale language model 4005, using tacit knowledge comments and the like. The text information generation unit 45 may generate text information using preset standard questions if no question text is entered, rather than using a method that does not use any questions at all.

[0136] The display control unit 13 of the terminal device 10 displays the text information related to comments C11 to C14 received from the three-dimensional image management server 40 on the display 106a.

[0137] <Action or process> <<Learning Phase (Model Update)>> First, with reference to Figure 9, we will explain the model update process in which the first tacit knowledge model 4004A learns data. Figure 9 is a sequence diagram showing an example of the model update process.

[0138] S1: The user enters a login operation into the terminal device 10. This login is for the captured image management server 20. The input reception unit 12 of the terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful.

[0139] Users log in to the captured image management server 20 and then to the three-dimensional image management server 40, but they may also log in to the three-dimensional image management server 40 first and then to the captured image management server 20.

[0140] S2: Upon successful login, the transmitting / receiving unit 11 of the terminal device 10 sends a request for the property selection screen 200 to the captured image management server 20.

[0141] S3: The transmitting / receiving unit 21 of the captured image management server 20 receives a request for the property selection screen 200. The screen generation unit 22 generates the property selection screen 200, and the transmitting / receiving unit 21 transmits the screen information of the property selection screen 200 to the terminal device 10.

[0142] S4: The transmitting / receiving unit 11 of the terminal device 10 receives screen information from the property selection screen 200. The display control unit 13 displays the property selection screen 200 (see Figure 11). The user inputs property identification information (for example, V0001, XX Building 2F-N, etc.) for the property whose live image they want to see on the displayed property selection screen 200. The input receiving unit 12 of the terminal device 10 receives the property identification information.

[0143] S5: The transmitting / receiving unit 11 of the terminal device 10 sends a request for live images to the captured image management server 20, specifying the property identification information.

[0144] S6: The transmitting / receiving unit 21 of the captured image management server 20 receives a request for live images, and the storage / reading unit 29 searches the captured image information management DB 2001 using the property identification information. The screen generation unit 22 of the captured image management server 20 generates a property management screen 210 that displays the live images, and the transmitting / receiving unit 21 transmits the screen information of the property management screen 210 to the terminal device 10.

[0145] Furthermore, the transmitting / receiving unit 21 transmits a live image of the object and an image request program to the terminal device 10 in response to a request for a live image. The image request program is, for example, a web application that enables the terminal device 10 to acquire three-dimensional image information. This web application is installed on the image capture management server 20 by the operator of the three-dimensional image management server 40 with the permission of the operator of the image capture management server 20. Alternatively, a URL containing the image request program may be sent to the terminal device 10. Since the web application acquires three-dimensional image information from the three-dimensional image management server 40, the terminal device 10 has the function of connecting to the three-dimensional image management server 40 and requesting or displaying three-dimensional image information.

[0146] S7: The transmitting / receiving unit 11 of the terminal device 10 receives the live image, the screen information of the property management screen 210, and the image request program. The display control unit 13 displays the property management screen 210 (see Figure 12). As a result, the property management information and the live image are displayed. The user requests three-dimensional image information of the property from the displayed property management screen 210 (by pressing the image acquisition button 213). The user can also arbitrarily change the viewpoint of the live image. In addition, the terminal device 10 can obtain the current shooting position information of the shooting device 5 from the captured image management server 20.

[0147] The input receiving unit 12 of the terminal device 10 receives an operation requesting three-dimensional image information of the object. Three-dimensional image information of the object refers to three-dimensional image information of items placed in the object, which is generated as a virtual space. The items are composed of 3D model shape information. Since the object has already been identified, the request for three-dimensional image information of the object may be sent to the three-dimensional image management server 40 without any operation from the user.

[0148] Here, the property management screen 210 includes a first display area 214 that displays live images acquired from the captured image management server 20, and a second display area 215 that displays three-dimensional image information of the items acquired from the three-dimensional image management server 40. At step S7, the first display area 214 displays property management information and live images, but nothing is displayed in the second display area 215.

[0149] S8: If the user is not logged in to the 3D image management server 40, the user enters a login operation into the terminal device 10. This login operation is directed to the 3D image management server 40. The input reception unit 12 of the terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful. The 3D image management server 40 may also skip the user's login operation by using, for example, single sign-on.

[0150] S9: The terminal device 10 executes an image request program to request three-dimensional image information. The transmitting / receiving unit 11 then sends the following to the three-dimensional image management server 40: the request for three-dimensional image information of the object selected by the user, the current shooting position information of the shooting device 5 (obtained from the image management server 20), and the field of view information specified by the user in step S7. The transmitting / receiving unit 11 may also send the URL of the image management server 20 to the three-dimensional image management server 40 so that the terminal device 10 can be redirected to the image management server 20. Three-dimensional image information of an object is an image of an object placed in an object formed as a virtual space. Since the object is composed of 3D model shape information, the terminal device 10 projects the 3D model shape of the object onto a two-dimensional plane to generate a planar image. The user can view any object while changing their viewpoint. The transmitting / receiving unit 11 may also send object management information obtained from the image management server 20 to the three-dimensional image management server 40. The image request program receives property management information from a web application connected to the image management server 20, for example, as a URL parameter.

[0151] S10: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives a request for three-dimensional image information of an object, specifying the object identification information of the object, the shooting position of the shooting device 5, and the field of view information. The storage / reading unit 49 searches the three-dimensional image information management DB 4001 using the object identification information and obtains the three-dimensional image information of each item. The processing unit 47 requests the screen generation unit 42 to generate a screen containing the three-dimensional image information of the object. The screen generation unit 42 generates the three-dimensional image information by placing a virtual camera at the position of the shooting position information and determining the field of view of the virtual camera based on the field of view information. The screen generation unit 42 generates a screen corresponding to the second display area 215 on which the three-dimensional image information is placed.

[0152] The transmitting / receiving unit 41 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The three-dimensional image information of each item included in the screen information is three-dimensional image information of all items included in the property placed on the property, and the user can change the viewpoint as they wish.

[0153] S11: The transmitting / receiving unit 11 of the terminal device 10 receives screen information for the screen corresponding to the second display area 215, and the display control unit 13 displays a three-dimensional image display screen 220 including the first display area 214 and the second display area 215 (see Figure 13). At step S11, only the three-dimensional image information of each item is displayed in the second display area 215, but the first display area 214 displays, for example, a live image, so that the live image and the three-dimensional image information of the item are displayed on the same screen from the same viewpoint. However, the viewpoint of both the live image and the three-dimensional image information can be changed.

[0154] Next, the user identifies an arbitrary item from the three-dimensional image information of the property (for example, by pressing it). The input reception unit 12 of the terminal device 10 accepts the operation to identify the item. The user can enlarge the arbitrary item or change the viewpoint. The user can also specify the field of view information. The terminal device 10 also obtains the current shooting position information of the shooting device 5 from the captured image management server 20. If the terminal device 10 is fixed, the shooting position information only needs to be obtained once. Once the user identifies an item, they can request a captured image of the item and the audio text associated with the captured image. The item may be identified, for example, by the coordinates clicked by the user, or the model ID may be identified by these coordinates.

[0155] Furthermore, users input comments (text information, audio) about the items into the terminal device 10, as explained in Figures 7 and 8. These comments are sometimes referred to as input information. These comments may constitute tacit knowledge comments. Additionally, the comments may include caption comments describing the items.

[0156] S12: When the user presses the information update button 226, the transmitting / receiving unit 11 of the terminal device 10 transmits information identifying the item (e.g., model ID), shooting location information, field of view information, and input information to the three-dimensional image management server 40.

[0157] S13: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives information that the information update button 226 has been pressed, information identifying the item (e.g., model ID), shooting location information, field of view information, and input information. The decision unit 48 decides to query the user to determine whether to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B. The transmitting / receiving unit 41 queries the terminal device 10 to determine whether to use captured images and voice text for model updating. The decision processing by the decision unit 48 will be explained in Figure 10 below.

[0158] S14: The transmitting / receiving unit 11 of the terminal device 10 receives the inquiry, and the display control unit 13 displays a message 227 on the three-dimensional image display screen 220 asking whether to use captured images and audio text for model updates (see Figure 14). The user confirms the message 227 and presses either the "Yes" button 228 or the "No" button 229. The input reception unit 12 receives the press. The transmitting / receiving unit 11 of the terminal device 10 sends "Yes" or "No" to the three-dimensional image management server 40. Note that the transition from Figure 13 to Figure 14 may be performed on the terminal device 10 side without communication with the three-dimensional image management server 40.

[0159] S15: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives a "needed" or "not needed" message, and the decision unit 48 determines whether a captured image is needed based on the "needed" or "not needed" message. Figure 9 illustrates the case where the image is needed. In order to acquire the captured image and audio text, the transmitting / receiving unit 41 transmits the shooting location information and field of view information to the terminal device 10. The three-dimensional image management server 40 transmits the shooting location information and field of view information in order to request the captured image management server 20 to provide a captured image taken from the same location and with the same field of view, along with the corresponding audio text.

[0160] S16: The transmitting / receiving unit 11 of the terminal device 10 receives a request for a captured image and audio text (shooting location information and field of view information). For example, the 3D image management server 40 notifies the terminal device 10 of the URL of the captured image management server 20 and redirects it. As a result, the transmitting / receiving unit 11 of the terminal device 10 specifies the shooting location information and field of view information and sends a request for a captured image and audio text to the captured image management server 20.

[0161] S17: The transmitting / receiving unit 21 of the captured image management server 20 receives a request for a captured image and audio text. The storage / reading unit 29 retrieves from the captured image information management DB 2001 a captured image that has the same location information as the shooting location information and is captured with the closest field of view information to the received field of view information, as well as the audio text associated with this captured image. It is expected that this captured image will show the same object as the three-dimensional image information. The transmitting / receiving unit 21 transmits the captured image and audio text to the terminal device 10. The captured image management server 20 may also capture from the latest live image using the shooting location information and field of view information.

[0162] S18: When the transmitting / receiving unit 11 of the terminal device 10 receives the captured image and audio text, it sends the captured image and audio text to the three-dimensional image management server 40. The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives the captured image and audio text as a response to the request in step S15. Upon receiving the captured image and audio text, the storage / reading unit 49 associates the captured image with the model ID identified in step S12 and saves it to the three-dimensional image information management DB 4001. The storage / reading unit 49 may also save the audio text.

[0163] S19: The processing unit 47 starts updating the first tacit knowledge model 4004A. First, the determination unit 43 obtains a caption comment identified by the model ID from the caption model 4003 and determines the degree of association between the caption comment and the comments included in the input information received in step S12. The determination unit 43 may determine the degree of association between the obtained caption comment and all the comments included in the input information received in step S12, or it may divide the comments included in the input information received in step S12 into multiple parts and determine the degree of association between each divided comment and the obtained caption comment.

[0164] S20: The update unit 46 updates the caption model 4003 by associating the comments deemed highly relevant in step S19 with the model ID as caption comments. The decision unit 48, having received a "required" response regarding the necessity of the captured image and audio text, decides to update the first tacit knowledge model 4004A. The update unit 46 updates the first tacit knowledge model 4004A using the comments and audio text deemed less relevant in step S19, along with the related three-dimensional image information of the item (identified in step S12) and the captured image, as training data. In other words, the correspondence between the three-dimensional image information of the item and the captured image, and the comments and audio text, is learned. Features are extracted from the three-dimensional image information of the item and the captured image using several feature extraction models suitable for images, such as CNNs. Features represent, for example, what is pictured where, or the content of the work being performed. Therefore, the first tacit knowledge model 4004A can learn the correspondence between three-dimensional image information and captured image features of items and comments and audio text.

[0165] Note that both comments and audio text are not necessarily required; the first tacit knowledge model 4004A can be updated even without comments.

[0166] Figure 10 is a flowchart illustrating how the decision unit 48 determines whether to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B. First, in Figure 10(a), the decision unit 48 determines whether it has received a message from the terminal device 10 indicating that a captured image and audio text are required (S301).

[0167] If the decision in step S301 is Yes, the decision unit 48 decides to update the first tacit knowledge model 4004A (S302).

[0168] If the decision in step S301 is No, the decision unit 48 decides to update the second tacit knowledge model 4004B (S303).

[0169] In addition, in Figure 10(b), the determination unit 48 determines whether the question sentence contained in the input information (voice, text) received from the terminal device 10 relates to a captured image of an item (S304).

[0170] If the decision in step S304 is Yes, the decision unit 48 decides to update the first tacit knowledge model 4004A (S305).

[0171] If the decision in step S304 is No, the decision unit 48 decides to update the second tacit knowledge model 4004B (S306).

[0172] Although Figures 10(a) and (b) illustrate the example of updating a model, this principle can also be applied to selecting a model used to generate text information.

[0173] <Screen example> Figure 11 shows an example of a property selection screen 200 for entering property identification information. The property selection screen 200 has a property identification information input field 201 and a search button 202. When the user enters the property identification information in the property identification information input field 201 and presses the search button 202, the list of room numbers shown in Figure 12 is displayed on the property management screen 210.

[0174] Figure 12 shows an example of a property management screen 210. The property management screen 210 includes a first display area 214 that displays information about items acquired from the captured image management server 20, and a second display area 215 that displays three-dimensional image information of items acquired from the three-dimensional image management server 40. The first display area 214 is the area other than the second display area 215. The first display area 214 includes a list 211 of room numbers belonging to the property identified by the property identification information and a live image 251. The live image 251 is a real-time video. Depending on the property, the room number may not be displayed, and the user may transition from the property selection screen 200 in Figure 11 to Figure 13 to display the three-dimensional image information of the property. The user selects the room number for which they want to display the three-dimensional image information using the mouse cursor 212. When the user presses the image acquisition button 213, the three-dimensional image display screen 220 is displayed.

[0175] The second display area 215 is an area other than the first display area 214, and its display may be implemented by a program on a web application, such as an iframe.

[0176] Figure 13 shows an example of a three-dimensional image display screen 220. The three-dimensional image display screen 220 includes a first display area 214 and a second display area 215. A live image 251 is displayed in the first display area 214 of the three-dimensional image display screen 220. The live image 251 is a real-time video.

[0177] The second display area 215 of the three-dimensional image display screen 220 displays three-dimensional image information 222. In the initial state, three-dimensional image information 222 is displayed with the same shooting position and field of view as the live image 251. This shooting position and field of view can be those specified by the user for the live image 251, or they can remain at the initial state. Since the three-dimensional image information 222 is an image projected from a three-dimensional model, the user can change the field of view information.

[0178] Furthermore, the user can select the item for which they want to display the captured image and audio text using the mouse cursor 212 from the three-dimensional image information 222. This determines the coordinates of the item as information to identify the item. Also, as the user changes the viewpoint or zooms in, the field of view of the three-dimensional image information 222 is determined. For example, the three-dimensional image information 223 of a table can be zoomed in and displayed. The shooting position of the shooting device 5 is also obtained from the captured image management server 20. When the user presses the information update button 226, the first tacit knowledge model 4004A is updated. The information display button 225 is used to display text information generated based on tacit knowledge comments, as will be described later.

[0179] Furthermore, since the live image 251 is a wide-field image, the user can change the field of view information. The user may also specify the field of view of the live image 251 in order to identify the item from which to acquire the captured image and audio text. In this case as well, the three-dimensional image management server 40 can identify the item selected by the user based on the position and field of view information of the shooting device 5. However, if it is three-dimensional image information 222, the terminal device 10 can uniquely identify the item by the coordinates of the mouse pointer relative to the 3D model.

[0180] In Figure 13, the area (floor area) of 224 is displayed as information about the property. The area (floor area) of 224 may be a measured value or may be included in the property management information.

[0181] As shown in Figure 13, the terminal device 10 was able to display the live image 251 managed by the captured image management server 20 and the three-dimensional image information 222 of the property managed by the three-dimensional image management server 40 on a single screen. Users can view the live image 251 while checking the three-dimensional image information 222 of the property. Users can also change the field of view information to compare the two.

[0182] Furthermore, the first display area 214 of the three-dimensional image display screen 220 displays the area (floor area) 224 as information about the property. The second display area 215 displays input information 241, which states, "This table has an unstable center of gravity, so it is best not to place anything weighing more than 50 kg on it." The three-dimensional image management server 40 can update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B using this input information 241 and voice text. Note that the area (floor area) 224 as information about the property can also be used as a caption comment.

[0183] When the user presses the information update button 226, the message 227 shown in Figure 14 pops up. Subsequently, the first tacit knowledge model 4004A or the second tacit knowledge model 4004B is updated. Similarly, when the information display button 225 is pressed, the message 237 shown in Figure 17 is displayed, and then the text information generated based on the tacit knowledge comment is displayed.

[0184] When the user presses the information update button 226, an update request for the tacit knowledge model is sent to the three-dimensional image management server 40. A message 227 is then displayed asking whether to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B.

[0185] Figure 14 shows message 227, which is displayed as a pop-up on the three-dimensional image display screen 220. Message 227 asks the user whether or not to use captured images for model updates. If captured images are used, audio text is also used, but the user may be able to set it to use only one or the other. The user presses the Yes button 228 if they want to update the first tacit knowledge model 4004A using captured images, and the No button 229 if they want to update the second tacit knowledge model 4004B without using captured images. One criterion for making this decision is whether the input information 241 is specific to this item or common to all items in the same category.

[0186] <Inference Phase (Generation of Text Information)> Next, with reference to Figure 15, the text information generation process using the first tacit knowledge model 4004A will be explained. Figure 15 is a sequence diagram showing an example of the text information generation process using the first tacit knowledge model 4004A. Note that the explanation of Figure 15 will mainly focus on the differences from Figure 9. The processes in steps S31 to S42 are the same as in Figure 9. However, in step S41, the user entered a question 234 about the item and pressed the information display button 225 (see Figure 16).

[0187] S43: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives information that the information display button 225 has been pressed, information identifying the item (shooting location information, field of view information), and input information. The decision unit 48 decides to query the user to determine whether to use the first tacit knowledge model 4004A or the second tacit knowledge model 4004B to generate the text information. The transmitting / receiving unit 41 queries the terminal device 10 to determine whether to use the captured image to generate the text information.

[0188] S44: The transmitting / receiving unit 11 of the terminal device 10 receives the inquiry, and the display control unit 13 displays a message 237 on the three-dimensional image display screen 220 asking whether or not to use the captured image to generate text information (see Figure 17). The user confirms the message 237 and presses either the Yes button 238 or the No button 239. The criteria for this decision will be described later. The input reception unit 12 receives the press. The transmitting / receiving unit 11 of the terminal device 10 sends Yes or No to the three-dimensional image management server 40.

[0189] S45: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives a "needed" or "not needed" message, and the decision unit 48 determines whether a captured image is needed based on the "needed" or "not needed" message. Figure 15 illustrates the case where the image is needed. In order to acquire the captured image and audio text, the transmitting / receiving unit 41 transmits the shooting location information and field of view information to the terminal device 10. The three-dimensional image management server 40 transmits the shooting location information and field of view information in order to request the captured image and audio text from the captured image management server 20.

[0190] The subsequent steps S46 to S48 can be carried out in the same manner as in Figure 9.

[0191] S49: When the transmitting / receiving unit 41 of the three-dimensional image management server 40 receives the captured image and audio text, the storage / reading unit 49 stores the captured image in the three-dimensional image information management DB 4001, associating it with the model ID. The processing unit 47 also requests the text information generation unit 45 to generate text information. Since it has been decided to generate text information using the captured image and audio text, the text information generation unit 45 obtains tacit knowledge comments corresponding to the three-dimensional image information of the item and the captured image from the first tacit knowledge model 4004A. The first tacit knowledge model 4004A can extract features of the three-dimensional image information of the item and the captured image, and identify comments, or audio text and comments, that correspond to these features. The tacit knowledge model extracts the comments, or audio text and comments, as tacit knowledge comments.

[0192] S50: Next, the text information generation unit 45 obtains text information created by the large-scale language model using the tacit knowledge comments, input information (question text), and speech text. The large-scale language model 4005 can use these to generate more detailed text information. The text information generation unit 45 may also convert the speech information contained in the input information (question text) into text information, and the text information generated by the text information generation unit 45 may be either speech information or text information.

[0193] The text information generation unit 45 may generate text information without using any speech text or question text. Alternatively, the text information generation unit 45 may generate fixed questions internally within the system and use these fixed questions. In this case, the question text is not visible to the user. Or, the text information generation unit 45 may generate fixed questions internally within the system, display these fixed questions on the display unit for the user to select, and then use the selected question.

[0194] While speech-to-text is not strictly required, using speech-to-text to generate text information from the large-scale language model 4005 allows for more detailed information about an item. For example, if the speech-to-text includes the extent of damage to the item, it is possible to generate text information that includes appropriate responses based on the extent of the damage.

[0195] S51: The processing unit 47 associates the three-dimensional image information, captured image, and text information of the item corresponding to the model ID (identified by information that identifies the item) and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to a second display area 215 that includes the three-dimensional image information, captured image, and further displays the generated text information.

[0196] The screen generation unit 42 may perform an update process that adds only text information to the screen corresponding to the second display area 215. The transmitting / receiving unit 41 of the three-dimensional image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmitting / receiving unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the second display area 215 transmitted from the three-dimensional image management server 40.

[0197] S52: The display control unit 13 of the terminal device 10 displays a text display screen 230 including a first display area 214 and a second display area 215 (see Figure 18). Three-dimensional image information, captured images, and text information are displayed in the second display area 215. The conversion unit 15 may convert the received text information into audio information, and the audio control unit 14 may play the converted text information through the speaker 109a. Alternatively, if the received text information is audio information, the text information may be played through the speaker 109a, or the conversion unit 15 may convert the received text information into character information and display the converted text information on the display 106a.

[0198] <<Example of the inference phase screen>> The screen displayed by the terminal device 10 during the inference phase is the same as in Figures 11 to 13, but in the three-dimensional image display screen 220 of Figure 13, the user enters input information (question text) and presses the information display button 225.

[0199] Figure 16 shows an example of a three-dimensional image display screen 220 in the inference phase (an example of the first display screen). The property display screen 240 includes a first display area 214 and a second display area 215. Figure 16 has the same configuration as Figure 13, but the user has entered a question 234 as input information. The user enters the input information (question 234) by pressing the three-dimensional image information 223 of the table. The input information (question 234) is displayed in the second display area 215, corresponding to the three-dimensional image information 223 of the table. For example, the question 234 in Figure 16 is "There is a scratch on the table, what should I do?" Along with such a question 234, the user presses the information display button 225 to request the generation of text information using the tacit knowledge model.

[0200] Figure 17 shows message 237, which is displayed as a pop-up on the three-dimensional image display screen 220. Message 237 asks the user whether to use the captured image to generate text information. The user presses the "Yes" button 238 if they want to generate text information using the captured image, or the "No" button 239 if they want to generate text information without using the captured image.

[0201] The user's decision-making criteria will be explained. As described above, the first tacit knowledge model 4004A and the second tacit knowledge model 4004B have the following differences. • The first tacit knowledge model 4004A can generate expert knowledge and text information specific to particular items. • The second tacit knowledge model 4004B can generate expert knowledge and general text information applicable to similar (same-category) items in general.

[0202] For example, if a specific item is a turbo chiller used by a customer, then because it is an item owned by the customer, there is an accumulation of inspection history and know-how regarding that item. The user decides to use this information and the first tacit knowledge model 4004A to generate highly accurate text information (answers) specifically for that particular item.

[0203] On the other hand, while such highly accurate information does not exist for all similar items (refrigeration machines in the category of turbo chillers), specialized knowledge about turbo chillers in general does exist. Therefore, in this case, it is thought that the user would use the second tacit knowledge model 4004B to generate general text information (answers) applicable to all similar items. Thus, the user can decide whether or not to use captured images to generate text information depending on how detailed the information required.

[0204] Furthermore, the decision unit 48 may decide whether to use a captured image to generate text information automatically (without confirming with the user) or semi-automatically (by recommending it to the user and asking for confirmation), rather than the user making the decision. For example, if the question sentence included in the input information (voice, text) concerns a captured image of an item, the decision unit 48 may decide to use the first tacit knowledge model 4004A because the second tacit knowledge model 4004B may not be able to generate appropriate text information.

[0205] Furthermore, there may be three or more types of tacit knowledge models, not just two, and users may be able to select the tacit knowledge model they prefer.

[0206] Figure 18 shows an example of text information displayed on the text display screen 230. The text display screen 230 includes a first display area 214 and a second display area 215. In Figure 18, the user specified a table and requested tacit knowledge comments, so the three-dimensional image information 223 of the table is displayed.

[0207] In Figure 18, the captured image 252 is displayed in the second display area 215. The captured image 252 is extracted from the captured image management server 20 if its field of view is close to that of the three-dimensional image information 223 in the table. The captured image 252 is obtained from the captured image information management table. If there are multiple captured images with the same field of view in the captured image information management table, this captured image 252 is the most recent captured image. Alternatively, multiple captured images may be displayed in reverse chronological order.

[0208] Text information 235 is "If the scratch is less than 1 mm deep, it will be repaired with paint. If it is 1 mm or deeper, it will be polished." Text information 235 is displayed in the second display area 215, corresponding to the table's three-dimensional image information 223. Text information 235 is generated by the large-scale language model 4005 from tacit knowledge comments, voice text, and question sentences. When the first tacit knowledge model 4004A detects, for example, that there is a scratch in the captured image of an item, it outputs a tacit knowledge comment regarding the scratch on this item. This tacit knowledge comment, a question sentence about the scratch, and voice text identifying the current state of the scratch are input to the large-scale language model 4005, so that it can generate appropriate text information in response to the question about the current scratch.

[0209] <Effects of generating text information using captured images> This invention will explain the effects of generating text information using captured images.

[0210] 1. Comparative Example 1 (Using a General Large-Scale Language Model) Question: "How should I repair the cracks?" Tacit comment: Tape or filler can be used.

[0211] 2. Comparative Example 2 (Case of learning from three-dimensional image information) • Learning Phase Input image: Three-dimensional image information Comment: Use tape for wide cracks and filler for narrow cracks. • Inference Phase Input image: 3D image display only Implicit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter.

[0212] 3. This embodiment (three-dimensional image information, captured image, audio text) • Learning Phase Input images: Three-dimensional image information and captured images Audio text: Applying tape to the corners may cause cracking. • Inference Phase Input images: Three-dimensional image information and captured images Question: "How should I repair the cracks?" Implicit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter, but be careful when applying tape to corners as it may cause cracks to form. In other words, "Be careful when applying tape to corners as it may cause cracks to form" is the effect of learning from the captured image and audio text.

[0213] 4. This embodiment (three-dimensional image information, captured image, audio text, input information) • Learning Phase Input images: Three-dimensional image information and captured images Audio text: Applying tape to the corners may cause cracking. Input information: Use tape for wide cracks and filler for narrow cracks. • Inference Phase Input images: 3D image display and captured images Question: "How should I repair the cracks?" Implicit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter, but please be careful when applying tape to corners as cracks may occur. In other words, "Please be careful when applying tape to corners as cracks may occur" is the effect of learning from the captured image and audio text.

[0214] <Example of a 3D image management server acquiring captured images and audio / text from a captured image management server> In Figures 9 and 15, the terminal device 10 acquires the captured image and audio text from the captured image management server 20, while the three-dimensional image management server 40 acquires them from the terminal device 10. However, it is also possible for the three-dimensional image management server 40 to directly acquire the captured image and audio text from the captured image management server 20.

[0215] Figure 19 is a sequence diagram showing an example of the process in which the 3D image management server 40 communicates with the captured image management server 20 to update the model. While Figure 19 mainly explains the differences from Figure 9, the sequence diagram in Figure 15 can be similarly modified. First, the processes in steps S1 to S14 are the same as in Figure 9.

[0216] S21: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives whether or not the captured image needs to be used. If it does, the processing unit 47 requests the transmitting / receiving unit 41 to request the captured image and audio text. The transmitting / receiving unit 41 calls the API of the captured image management server 20 and sends a request for the captured image and audio text to the captured image management server 20, specifying the model ID. Note that if the information identifying the item is the model ID, a search of the three-dimensional image information management DB 4001 is unnecessary.

[0217] S22: The transmitting / receiving unit 21 of the captured image management server 20 receives a request for a captured image and audio text. The storage / reading unit 29 retrieves from the captured image information management DB 2001 a record with the same location information as the shooting location information, a captured image captured with the closest field of view information to the received field of view information, and the audio text associated with this captured image. It is expected that this captured image will show the same object as the three-dimensional image information. The transmitting / receiving unit 21 sends the captured image and audio text to the three-dimensional image management server 40.

[0218] The subsequent processing can be the same as in Figure 9. Also, in Figure 19, the process by which the 3D image management server 40 acquires captured images and audio text from the captured image management server 20 was explained using a sequence diagram during model update as an example, but the process is the same when generating text information as in Figure 15.

[0219] <When a second tacit knowledge model is used for learning or inference> <<Learning Phase (Model Update)>> Next, with reference to Figure 20, we will describe the model update process in which the second tacit knowledge model 4004B learns data. Figure 20 is a sequence diagram showing an example of the model update process. Note that the explanation of Figure 20 will mainly focus on the differences from Figure 9. The processes in steps S61 to S74 can be the same as in Figure 9.

[0220] S75: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives a "needed" or "not needed" message, and the decision unit 48 determines whether a captured image is needed based on the "needed" or "not needed" message. Figure 20 illustrates the case where the image is not needed. Since a captured image is not needed, the process of the three-dimensional image management server 40 acquiring the captured image and audio text from the captured image management server 20 is not performed. The relevance determination process can be the same as in Figure 9.

[0221] S76: The update unit 46 updates the caption model 4003 by associating the comments deemed highly relevant in step S75 with the model ID as caption comments. The update unit 46 also updates the second tacit knowledge model 4004B using the comments deemed less relevant in step S75 and the related three-dimensional image information of the item (identified in step S72) as training data. In other words, the correspondence between the three-dimensional image information of the item and the comments is learned (captured images and audio text are not learned). Features are extracted from the three-dimensional image information of the item using several feature extraction models suitable for images, such as CNNs. Features represent, for example, what is pictured where, or the content of the work being performed. Therefore, the tacit knowledge model can learn the correspondence between the features of the three-dimensional image information of the item and the comments.

[0222] The screen displayed by the terminal device 10 during the learning phase may be the same as that shown in Figures 11 to 14.

[0223] <Inference Phase (Generation of Text Information)> Next, with reference to Figure 21, the text information generation process using the second tacit knowledge model 4004B will be explained. Figure 21 is a sequence diagram showing an example of the text information generation process using the second tacit knowledge model 4004B. Note that the explanation of Figure 21 may mainly focus on the differences from Figure 15. The processing in steps S81 to S94 can be the same as in Figure 15.

[0224] S95: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives a "needed" or "not needed" message, and the decision unit 48 determines whether a captured image is needed based on the "needed" or "not needed" message. Figure 21 illustrates the case where the image is not needed. Since a captured image is not needed, the three-dimensional image management server 40 does not perform the process of acquiring the captured image and audio text from the captured image management server 20.

[0225] Next, the processing unit 47 requests the text information generation unit 45 to generate text information. The text information generation unit 45 obtains tacit knowledge comments corresponding to the three-dimensional image information of the item from the second tacit knowledge model 4004B. The second tacit knowledge model 4004B can extract features of the three-dimensional image information of the item and identify comments (excluding audio text) corresponding to these features. The tacit knowledge model extracts the comments as tacit knowledge comments. S96: Next, the text information generation unit 45 obtains text information created by the large-scale language model 4005 using the tacit knowledge comments and input information (question text). The large-scale language model 4005 can use these to generate more detailed text information. The text information generation unit 45 may also convert the audio information contained in the input information into text information, and the text information generated by the text information generation unit 45 may be either audio information or text information.

[0226] The text information generation unit 45 may generate text information without using any question text. Alternatively, the text information generation unit 45 may generate fixed questions internally within the system and use those fixed questions. In this case, the question text is not visible to the user. Or, the text information generation unit 45 may generate fixed questions internally within the system, display the fixed questions on the display unit for the user to select, and use the selected question.

[0227] The subsequent steps can be the same as in Figure 15.

[0228] <<Screen example>> In the inference phase, among the screens displayed by the terminal device 10, the property selection screen 200 may be the same as in Figure 11, and the property management screen 210 may be the same as in Figure 12. Also, the three-dimensional image display screen 220 will be the same as in Figures 16 and 17. On the other hand, in the processing of Figure 21, the text information 235 of the text display screen 230 in Figure 18 will be different.

[0229] Figure 22 shows a text display screen 230 containing text information generated based on the second tacit knowledge model 4004B. This text information 236 reads, "Scratches can be repaired with paint or polishing." The text information 236 is displayed in the second display area 215, corresponding to the three-dimensional image information 223 of the table. The text information 236 was generated by the large-scale language model 4005 from the tacit knowledge comment generated by the second tacit knowledge model 4004B and the input information (question text). Therefore, even if there are scratches on the actual item, the tacit knowledge comment does not reflect the captured image. Also, the large-scale language model 4005 does not use speech text to generate the text information.

[0230] Therefore, comparing the text information 236 in Figure 22 with the text information 235 in Figure 18, text information 236 is general text information about table scratches and is less detailed than text information 235. However, text information 236 is highly versatile information regarding table scratches.

[0231] <Multimodal> Several examples of combinations of input information and tacit knowledge comments are described. While the model described above assumed a large-scale language model, this embodiment can utilize a multimodal model that takes multiple data formats (images, text, gestures, etc.) as input and outputs them in a predetermined data format. • When the input information is a string, and content other than text information is generated as an implicit knowledge comment. Enter a string of text to generate an image. Enter text to generate a video. Enter text to generate speech. Enter a string of text to generate a 3D model. • When input information includes both strings and non-strings, and text information is generated as an implicit knowledge comment. Enter an image and text to generate text information. Enter a 3D model and text to generate text information. Input speech and text to generate text information. • When input information includes both text and non-textual content, and non-textual content is generated as an implicit knowledge comment. Enter an image and text to generate an image. Enter a video and text to generate a video. Enter a 3D model and text to generate a 3D model. Input speech and text to generate speech. <Main effects> According to this embodiment, users can choose between a first tacit knowledge model 4004A, which is trained using captured images and audio text, and a second tacit knowledge model 4004B, which is trained without using captured images and audio text. In other words, the three-dimensional image management server 40 can use the first tacit knowledge model 4004A to generate detailed text information for items for which captured images and audio text have been stored, and can use the second tacit knowledge model 4004B to generate highly versatile text information for items in the same category in general.

[0232] [Second Embodiment] This embodiment describes the updating and inference of the tacit knowledge model when a user logs in to the three-dimensional image management server 40. When a user logs in directly to the three-dimensional image management server 40, it is unclear whether the user has the authority to log in to the captured image management server 20. However, the first tacit knowledge model 4004A is updated using captured images and audio text. Therefore, it is undesirable to allow the use of the first tacit knowledge model 4004A when a user who does not have the authority to log in to the captured image management server 20 logs in to the three-dimensional image management server 40. Accordingly, in this embodiment, when a user logs in directly to the three-dimensional image management server 40, only the updating of the second tacit knowledge model 4004B and the generation of text information are permitted.

[0233] In this embodiment, the hardware configuration diagram shown in Figure 2 and the functional block diagram shown in Figure 3, as described in the above embodiment, can be used as references.

[0234] <Action or process> <<Learning Phase (Model Update)>> Referring to Figure 23, we will describe the model update process in which the second tacit knowledge model 4004B learns data. Figure 23 is a sequence diagram showing an example of the model update process.

[0235] S101: The user enters a login operation into terminal device 10. This login is for the three-dimensional image management server 40. The input reception unit 12 of terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful.

[0236] S102: Upon successful login, the transmitting / receiving unit 11 of the terminal device 10 sends a request for the property selection screen 200 to the three-dimensional image management server 40.

[0237] S103: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives a request for the property selection screen 200. The screen generation unit 42 generates the property selection screen 200, and the transmitting / receiving unit 41 transmits the screen information of the property selection screen 200 to the terminal device 10.

[0238] S104: The transmitting / receiving unit 11 of the terminal device 10 receives screen information from the property selection screen 200. The display control unit 13 displays the property selection screen 200 (see Figure 11). The user enters property identification information (e.g., V0001) on the displayed property selection screen 200. The input receiving unit 12 of the terminal device 10 receives the property identification information.

[0239] S105: The transmitting / receiving unit 11 of the terminal device 10 sends a request for three-dimensional image information of the property to the three-dimensional image management server 40, specifying the property identification information. Since the terminal device 10 is not logged into the captured image management server 20, the property management screen 210 in Figure 12 is not displayed.

[0240] S106: The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives a request, and the storage / reading unit 49 searches the three-dimensional image information management DB 4001 using the object identification information. The storage / reading unit 49 acquires the image information of each item. The screen generation unit 42 generates a screen corresponding to the second display area 215, which displays the three-dimensional image information of the items in association with each other. The transmitting / receiving unit 41 transmits the three-dimensional image information corresponding to the screen corresponding to the second display area 215 to the terminal device 10. The three-dimensional image information of each item is the three-dimensional image information of the item placed in the object identified by the object identification information. Since the items are composed of 3D model shape information, the terminal device 10 projects the three-dimensional model shape of the item onto two dimensions to generate a planar image. Users can view any item while changing their viewpoint.

[0241] S107: The transmitting / receiving unit 11 of the terminal device 10 receives three-dimensional image information of the screen corresponding to the second display area 215, and the display control unit 13 displays the property display screen 260 including the second display area 215 (see Figure 25). In this embodiment, since the terminal device 10 is not logged into the captured image management server 20, the list of items placed in the property is not displayed. However, the screen generation unit 42 may use the three-dimensional image information management DB 4001 managed by the three-dimensional image management server 40 to display information equivalent to the item list. Next, the user identifies an arbitrary item from the three-dimensional image information of the property. The input reception unit 12 of the terminal device 10 accepts the operation to identify an item. The item may be identified, for example, by the coordinates clicked by the user, or the model ID may be identified by these coordinates.

[0242] The user inputs comments (text information, audio) about the item into the terminal device 10, as explained in Figures 7 and 8. These comments are sometimes referred to as input information. These comments may constitute tacit knowledge comments. Furthermore, the comments may include caption comments describing the item.

[0243] S108: When the user presses the information update button 226, the transceiver 11 of the terminal device 10 transmits information indicating that the information update button 226 has been pressed, information for specifying an article, and input information to the three-dimensional image management server 40. The transceiver 41 of the three-dimensional image management server 40 receives these. The determination unit 48 determines which of the first tacit knowledge model 4004A and the second tacit knowledge model 4004B to update. Since the determination unit 48 finds that the login in step S101 is not a login using the image request program distributed from the photographed image management server 20 (since it is found that it is a direct login to the three-dimensional image management server 40), it determines to update the second tacit knowledge model 4004B. The determination in step S108 will be described with reference to FIG. 24.

[0244] The processing of subsequent steps S109 and S110 may be the same as that in FIG. 20. That is, the second tacit knowledge model 4004B is updated.

[0245] FIG. 24 is a flowchart for the determination unit 48 to determine whether to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B. In FIG. 24, the determination unit 48 determines whether it has logged in to the three-dimensional image management server 40 via the photographed image management server 20 (S311). The determination unit 48 can determine whether it has logged in to the three-dimensional image management server 40 via the photographed image management server 20 based on whether it is a login using the image request program distributed from the photographed image management server 20.

[0246] If the determination in step S311 is Yes, the determination unit 48 determines to update the first tacit knowledge model 4004A (S312).

[0247] If the determination in step S311 is No, the determination unit 48 determines to update the second tacit knowledge model 4004B (S313).

[0248] Note that although FIG. 24 has been described by taking the update of the model as an example, it can also be applied to the selection of the model used for generating text information.

[0249] <<Screen example>> The property selection screen 200 displayed by the terminal device 10 during the learning phase may be the same as in Figure 11. Figure 12 is not displayed because it is a screen generated by the captured image management server 20. The property display screen 260 of this embodiment will be described based on Figure 25.

[0250] Figure 25 shows the item display screen 260 of this embodiment. The item display screen 260 includes a second display area 215. Comparing Figure 25 with Figure 13, the item list 221 is not displayed. This is because the terminal device 10 logged directly into the three-dimensional image management server 40, and therefore the item list 221 managed by the captured image management server 20 is not displayed.

[0251] Users can change the viewpoint and field of view of the three-dimensional image information 222. While captured images and audio text are not acquired based on the viewpoint and field of view, the three-dimensional image information 222 identified by the viewpoint and field of view can be used for training.

[0252] <Inference Phase (Generation of Text Information)> Next, with reference to Figure 26, the text information generation process using the second tacit knowledge model 4004B will be explained. Figure 26 is a sequence diagram showing an example of the text information generation process using the second tacit knowledge model 4004B, where the terminal device 10 directly logs into the three-dimensional image management server 40. Note that the explanation of Figure 26 may mainly focus on the differences from Figure 23. The processing in steps S121 to S128 can be the same as in Figure 23. However, in step S127, the user inputs a question about an item (see Figure 27).

[0253] S129: The transmitting / receiving unit 11 of the terminal device 10 transmits to the three-dimensional image management server 40 that the information display button 225 has been pressed, information identifying the item, and input information (question text). The transmitting / receiving unit 41 of the three-dimensional image management server 40 receives these. The decision unit 48 decides whether to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B. The decision unit 48 determines that the login in step S121 is not a login using an image request program distributed from the captured image management server 20 (it is a direct login to the three-dimensional image management server 40), and therefore decides to generate text information using the second tacit knowledge model 4004B.

[0254] The subsequent processing can be the same as steps S95 to S98 in Figure 21. That is, text information is generated by the second tacit knowledge model 4004B and the large-scale language model 4005.

[0255] <<Screen example>> The property selection screen 200 displayed by the terminal device 10 during the inference phase may be the same as in Figure 11. Figure 12 is a screen generated by the captured image management server 20 and is therefore not displayed. The property display screen 260 during the inference phase is as shown in Figure 27, and the text display screen 270 during the inference phase is as shown in Figure 28.

[0256] Figure 27 shows the property display screen 260 when the terminal device 10 logs directly into the three-dimensional image management server 40 (an example of the second display screen). Comparing Figure 27 with Figure 16, the item list 221 and live image 251 are not displayed. This is because the terminal device 10 logged directly into the three-dimensional image management server 40, and therefore the item list 221 and live image 251, which are managed by the captured image management server 20, are not displayed.

[0257] Furthermore, users can change the viewpoint and field of view of the three-dimensional image information 222. While captured images and audio text are not acquired based on the viewpoint and field of view, the three-dimensional image information 222 identified by the viewpoint and field of view may be used to generate text information.

[0258] Figure 28 shows the text display screen 270 of this embodiment. Comparing Figure 28 with Figure 22, the item list 221 and live image 251 are not displayed. This is because the terminal device 10 directly logged into the three-dimensional image management server 40. Also, in this embodiment, since the text information 236 is generated based on the second tacit knowledge model 4004B, the same text information 236 as in Figure 22 is displayed.

[0259] <Main effects> According to this embodiment, if a user logs in directly to the three-dimensional image management server 40, the provision of text information by the first tacit knowledge model 4004A, which has learned captured images and audio text, can be restricted. Even in this case, the three-dimensional image management server 40 can still provide the user with text information by the second tacit knowledge model 4004B, which has not learned captured images and audio text. If a user logs in to the three-dimensional image management server 40 via the captured image management server 20, the first tacit knowledge model 4004A, which has learned captured images and audio text, can still be provided to the user with text information.

[0260] [Third Embodiment] In this embodiment, an information processing system 100 in which two terminal devices each generate text information will be described.

[0261] <Example System Configuration> Figure 29 is an overall configuration diagram of the information processing system 100 according to the embodiment. The explanation of Figure 29 will mainly describe the differences from Figure 1. As shown in Figure 29, the information processing system 100 has terminal devices 10A and 10B. Any of the terminal devices 10A and 10B will simply be referred to as "terminal device 10". Users of terminal devices 10A and 10B are optional. For the sake of explanation, it will be assumed that terminal device 10A logs in to the captured image management server 20, and terminal device 10B logs in to the three-dimensional image management server 40. The functions of terminal devices 10A and 10B may be the same as in Figure 3.

[0262] The terminal device 10A (an example of the first terminal device) executes the processes described in the first embodiment, and the terminal device 10B (an example of the second terminal device) executes the processes described in the second embodiment. That is, the terminal device 10 in the first embodiment corresponds to the terminal devices 10A and 10B, respectively. The terminal device 10A performs model update and text information generation, and the terminal device 10B performs model update and text information generation.

[0263] <Main effects> In this way, the three-dimensional image management server 40 can appropriately use the first tacit knowledge model 4004A or the second tacit knowledge model 4004B according to the login path, whether the terminal device 10A logs in via the photographed image management server 20 or the terminal device 10B logs in directly to the three-dimensional image management server 40. Also, even if the terminal devices 10A and 10B log in to the three-dimensional image management server 40 in parallel (simultaneously), the first tacit knowledge model 4004A or the second tacit knowledge model 4004B can be appropriately used.

[0264] [Fourth Embodiment] In this embodiment, a three-dimensional image management server 40 that generates an image from a photographed image and text information will be described.

[0265] FIG. 30 is a functional configuration diagram illustrating an example of the functions of the three-dimensional image management server 40, the photographed image management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment. In the description of FIG. 30, mainly the differences from FIG. 3 will be described.

[0266] The three-dimensional image management server 40 in FIG. 30 further includes an image generation unit 51, and an image generation model 4006 is further constructed in the storage unit 4000 of the three-dimensional image management server 40. Other configurations may be the same as those in FIG. 3.

[0267] The image generation unit 51 is an example of image generation means, and is realized by an instruction from the CPU 401 shown in FIG. 2. It inputs text data or text data into the image generation model 4006 to generate image information.

[0268] Image generation model 4006 is a machine learning model (generative AI) that generates images from text data, or from text data and images. Image generation model 4006 is trained using training data that includes text data and images. This training data includes, for example, training text data, or text data and images as input, and ground truth images as output. For example, the image generation model 4006 may be trained so that the images it generates when inputting text data, or text data and images, approach the ground truth images included in the training data.

[0269] <Learning Phase> The processing in the learning phase may be the same as in Figure 19. In step S24, the update unit 46 updates the first tacit knowledge model 4004A to learn a correspondence between comments and voice text that were determined to have a low degree of relevance in step S23, and three-dimensional image information or captured images of the item. Alternatively, the update unit 46 updates the tacit knowledge model 4004 to learn a correspondence between comments, voice text, and three-dimensional image information (or captured images) of the item, and captured images (or three-dimensional image information) as output.

[0270] <Inference Phase (Generation of Text Information)> Figure 31 is an example sequence diagram showing the process of generating text and image information. Note that the explanation of Figure 31 mainly focuses on the differences from Figure 21. In Figure 31, step S96-1 is added.

[0271] S96-1: The image generation unit 51 inputs the captured image and the text information created by the large-scale language model to the image generation model 4006 to generate image information. The image generation unit 51 may also obtain the image information created by the image generation model 4006 using the text information created by the large-scale language model without using the captured image.

[0272] The memory / reading unit 49 stores (or overwrites) the text information created by the large-scale language model and the image information created by the image generation model 4006 in the three-dimensional image information management DB 4001, corresponding to the captured image stored in the DB 4001 in step S92.

[0273] S97: The processing unit 47 requests the screen generation unit 42 to display the three-dimensional image information of the item corresponding to the model ID, the generated image information, and the text information. The screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the three-dimensional image information of the item, the generated image information, and the text information. The transmitting / receiving unit 41 of the three-dimensional image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215 transmitted from the three-dimensional image management server 40.

[0274] <Example of the inference phase screen> Figure 32 shows the generated image information displayed on the text display screen 280. The explanation of Figure 32 mainly focuses on the differences from Figure 18.

[0275] The text display screen 280 in Figure 32 shows the generated image 261. The generated image 261 is not the captured image 252 described in Figure 18, but rather a generated image produced by the image generation model 4006 based on the captured image 252 and the text information 235. Therefore, the generated image 261 has a marker 263 that indicates the location of the wound.

[0276] <Main effects> In this way, the three-dimensional image management server 40 can generate image information based on an image generation model using captured images and text information.

[0277] [Other application examples] The present invention is not limited to the embodiments specifically disclosed above, and various modifications and changes are possible without departing from the scope of the claims. It goes without saying that the three-dimensional image management server 40 described in this embodiment is just one example, and there are various system configurations depending on the application and purpose.

[0278] For example, in this embodiment, we described an example in which an tacit knowledge model for industries such as civil engineering and construction answers a question, but the tacit knowledge model can be used in any industry where tacit knowledge is effective, such as healthcare, dental care, and investment decisions.

[0279] Furthermore, in this embodiment, the large-scale language model 4005 generates text information based on tacit knowledge comments, but it is also possible to use tacit knowledge comments as text information without using the large-scale language model 4005.

[0280] Furthermore, the first tacit knowledge model 4004A may also be one that takes three-dimensional image information, captured images, and audio text as inputs, and learns tacit knowledge comments by outputting the input information. In other words, it may take different forms of information, such as images and text, as inputs.

[0281] Furthermore, the three-dimensional image management server 40 may generate two sets of text information using not only one of the first tacit knowledge model 4004A or the second tacit knowledge model 4000B, but both. In other words, the three-dimensional image management server 40 may generate text information using at least one of the first tacit knowledge model 4004A or the second tacit knowledge model 4000B.

[0282] Furthermore, although this embodiment describes a client-server type information processing system 100, the functions of the three-dimensional image management server 40 may be installed as an application on the terminal device 10. In other words, users may use the functions of this embodiment in a standalone format.

[0283] Furthermore, the configuration examples shown in Figure 3 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the three-dimensional image management server 40. The present invention is not limited by the way the processing units are divided or their names. The processing of the three-dimensional image management server 40 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processing.

[0284] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0285] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the three-dimensional image management server 40 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0286] Furthermore, the three-dimensional image management server 40 can combine the disclosed processing steps in various ways. Each element of the three-dimensional image management server 40 may be combined into a single device or divided into multiple devices. Also, each processing performed by the three-dimensional image management server 40 may be performed by the terminal device 10.

[0287] <Mode> [Aspect 1] An information processing system including a three-dimensional image management server for managing three-dimensional image information of an object, and a terminal device capable of communicating with the three-dimensional image management server, The aforementioned three-dimensional image management server is A first model that has learned the correspondence between the three-dimensional image information of the object, a predetermined region image in the captured image obtained by photographing the object with a photography device, and the input information input to the terminal device, A second model that has learned the correspondence between the three-dimensional image information of the object and the input information input to the terminal device, The terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model, or The terminal device has a text information generation unit that generates text information about the object using the three-dimensional image information of the object selected by the terminal device and the second model, The aforementioned terminal device is The system has a display control unit that displays a display screen containing the aforementioned text information. Information processing system. [Aspect 2] The text information generation unit generates the text information based on the three-dimensional image information of the object selected by the terminal device, the predetermined region image, the input information received by the input receiving unit of the terminal device, and the first model, or, The information processing system according to Embodiment 1, which generates text information based on three-dimensional image information of the object selected by the terminal device, input information received by the input receiving unit of the terminal device, and the second model. [Aspect 3] The system further includes a captured image management server that manages the predetermined area image and is capable of communicating with the terminal device and the three-dimensional image management server. The information processing system according to embodiment 1 or 2, wherein the text information generation unit generates the text information based on the three-dimensional image information of the object, the predetermined region image transmitted by the captured image management server, and the first model. [Aspect 4] The aforementioned three-dimensional image management server is The terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model, or A determination unit that determines whether to generate text information about the object using the three-dimensional image information of the object selected by the terminal device and the second model, An information processing system according to any one of the embodiments 1 to 3. [Aspect 5] The determination unit determines whether to use the first model or the second model based on whether to use the predetermined region image received by the terminal device, or based on the voice or text input information received by the input receiving unit of the terminal device. The information processing system described in Embodiment 4. [Aspect 6] The aforementioned three-dimensional image management server is The first model is updated by learning the correspondence between the three-dimensional image information of the object and the predetermined region image and the input information input to the terminal device, or An update unit updates the second model by learning the correspondence between the three-dimensional image information of the object and the input information input to the terminal device, without using the predetermined region image. An information processing system according to any one of embodiments 1 to 5. [Aspect 7] The aforementioned determination unit, The first model is updated by learning the correspondence between the three-dimensional image information of the object and the predetermined region image and the input information input to the terminal device, or Without using the predetermined region image, the system decides whether to update the second model by learning the correspondence between the three-dimensional image information of the object and the input information input to the terminal device. The information processing system described in Embodiment 4. [Aspect 8] The display control unit, The input information is displayed on a first display screen that includes the captured image obtained from the captured image management server and the three-dimensional image information of the object obtained from the three-dimensional image management server, on a single screen, or The input information is displayed on a second display screen that includes three-dimensional image information of the object obtained from the three-dimensional image management server, without including the aforementioned captured image. The text information generation unit, Based on the input information displayed on the first display screen, the three-dimensional image information of the object, the predetermined region image, and the first model, the text information is generated, or, Based on the input information displayed on the second display screen, the three-dimensional image information of the object, and the second model, the text information is generated. The information processing system described in Embodiment 3. [Aspect 9] The text information generation unit, Based on the input information displayed on the first display screen, the three-dimensional image information of the object, the predetermined region image, and the first model, the text information is generated, or, Based on the input information displayed on the second display screen, the three-dimensional image information of the object, and the second model, the text information is generated. The information processing system described in aspect 8. [Aspect 10] The first model is updated by learning the correspondence between the three-dimensional image information of the object and the predetermined region image displayed on the first display screen and the input information, or, The system includes an update unit that updates the second model by learning the correspondence between the three-dimensional image information of the object displayed on the second display screen and the input information. The information processing system described in aspect 8 or 9. [Aspect 11] The aforementioned update unit is The first model is updated by learning the correspondence between the three-dimensional image information of the object and the predetermined region image displayed on the first display screen and the input information, or, The second model is updated by learning the correspondence between the three-dimensional image information of the object displayed on the second display screen and the input information. The information processing system described in aspect 10. [Aspect 12] The system has a first terminal device and a second terminal device that can communicate with the aforementioned three-dimensional image management server, The display control unit of the first terminal device displays a first display screen that includes the three-dimensional image information of the object acquired from the three-dimensional image management server and the predetermined area image acquired from the captured image management server. The display control unit of the second terminal device displays a second display screen that includes the three-dimensional image information of the object obtained from the three-dimensional image management server. The text information generation unit, The first terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model. The second terminal device receives the selection of the object and generates text information about the object using the second model. The information processing system described in Embodiment 3. [Aspect 13] The three-dimensional image information of the object is a two-dimensional projection of the object's three-dimensional model shape, and the three-dimensional image information of the object can be displayed by changing the viewpoint. An information processing system as described in any one of the descriptions in 1 to 12. [Aspect 14] The aforementioned image management server further manages audio and text, The text information generation unit generates the text information based on the three-dimensional image information of the object, the predetermined region image and audio text transmitted by the captured image management server, and the first model. The information processing system described in Embodiment 3. [Explanation of symbols]

[0288] 10 Terminal devices 20. Image management server 40. Three-dimensional image management server 100 Information Processing Systems [Prior art documents] [Patent Documents]

[0289] [Patent Document 1] Patent No. 6839123

Claims

1. An information processing system including a three-dimensional image management server for managing three-dimensional image information of an object, and a terminal device capable of communicating with the three-dimensional image management server, The aforementioned three-dimensional image management server is A first model that has learned the correspondence between the three-dimensional image information of the object, a predetermined region image in the captured image obtained by photographing the object with a photography device, and the input information input to the terminal device, A second model that has learned the correspondence between the three-dimensional image information of the object and the input information input to the terminal device, The terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model, or The terminal device has a text information generation unit that generates text information about the object using the three-dimensional image information of the object selected by the terminal device and the second model, The aforementioned terminal device is The system has a display control unit that displays a display screen containing the aforementioned text information. Information processing system.

2. The text information generation unit generates the text information based on the three-dimensional image information of the object selected by the terminal device, the predetermined region image, the input information received by the input receiving unit of the terminal device, and the first model, or, The information processing system according to claim 1, wherein the terminal device receives three-dimensional image information of the object selected, input information received by the input receiving unit of the terminal device, and the second model are used to generate the text information.

3. The system further includes a captured image management server that manages the predetermined area image and is capable of communicating with the terminal device and the three-dimensional image management server. The information processing system according to claim 1 or 2, wherein the text information generation unit generates the text information based on the three-dimensional image information of the object, the predetermined region image transmitted by the captured image management server, and the first model.

4. The aforementioned three-dimensional image management server is The terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model, or A decision unit that determines whether to generate text information about the object using the three-dimensional image information of the object selected by the terminal device and the second model, The information processing system according to claim 1, having the following features.

5. The determination unit determines whether to use the first model or the second model based on whether to use the predetermined region image received by the terminal device, or based on the voice or text input information received by the input receiving unit of the terminal device. The information processing system according to claim 4.

6. The aforementioned three-dimensional image management server is The first model is updated by learning the correspondence between the three-dimensional image information of the object and the predetermined region image and the input information input to the terminal device, or An update unit updates the second model by learning the correspondence between the three-dimensional image information of the object and the input information input to the terminal device, without using the predetermined region image. The information processing system according to claim 1, having the following features.

7. The aforementioned determination unit, The first model is updated by learning the correspondence between the three-dimensional image information of the object and the predetermined region image and the input information input to the terminal device, or Without using the predetermined region image, the system decides whether to update the second model by learning the correspondence between the three-dimensional image information of the object and the input information input to the terminal device. The information processing system according to claim 4.

8. The display control unit, The input information is displayed on a first display screen that includes the captured image acquired from the captured image management server and the three-dimensional image information of the object acquired from the three-dimensional image management server on a single screen, or The input information is displayed on a second display screen that includes three-dimensional image information of the object obtained from the three-dimensional image management server, without including the aforementioned captured image. The text information generation unit, Based on the input information displayed on the first display screen, the three-dimensional image information of the object, the predetermined region image, and the first model, the text information is generated, or, Based on the input information displayed on the second display screen, the three-dimensional image information of the object, and the second model, the text information is generated. The information processing system according to claim 3.

9. The text information generation unit, Based on the input information displayed on the first display screen, the three-dimensional image information of the object, the predetermined region image, and the first model, the text information is generated, or, Based on the input information displayed on the second display screen, the three-dimensional image information of the object, and the second model, the text information is generated. The information processing system according to claim 8.

10. The first model is updated by learning the correspondence between the three-dimensional image information of the object and the predetermined region image displayed on the first display screen and the input information, or, The system includes an update unit that updates the second model by learning the correspondence between the three-dimensional image information of the object displayed on the second display screen and the input information. The information processing system according to claim 8.

11. The aforementioned update unit is The first model is updated by learning the correspondence between the three-dimensional image information of the object and the predetermined region image displayed on the first display screen and the input information, or, The second model is updated by learning the correspondence between the three-dimensional image information of the object displayed on the second display screen and the input information. The information processing system according to claim 10.

12. The system has a first terminal device and a second terminal device that can communicate with the aforementioned three-dimensional image management server. The display control unit of the first terminal device displays a first display screen that includes the three-dimensional image information of the object acquired from the three-dimensional image management server and the predetermined area image acquired from the captured image management server. The display control unit of the second terminal device displays a second display screen that includes the three-dimensional image information of the object acquired from the three-dimensional image management server. The text information generation unit, The first terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model. The second terminal device receives the selection of the object and generates text information about the object using the second model. The information processing system according to claim 3.

13. The three-dimensional image information of the object is a two-dimensional projection of the object's three-dimensional model shape, and the three-dimensional image information of the object can be displayed by changing the viewpoint. The information processing system according to claim 1.

14. The aforementioned image management server further manages audio and text, The text information generation unit generates the text information based on the three-dimensional image information of the object, the predetermined region image and audio text transmitted by the captured image management server, and the first model. The information processing system according to claim 3.

15. A three-dimensional image management server that manages three-dimensional image information of an object that can communicate with a terminal device, A first model that has learned the correspondence between the three-dimensional image information of the object, a predetermined region image in the captured image obtained by photographing the object with a photography device, and the input information input to the terminal device, A second model that has learned the correspondence between the three-dimensional image information of the object and the input information input to the terminal device, The terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model, or The terminal device has a text information generation unit that generates text information about the object using the three-dimensional image information of the object selected by the terminal device and the second model, The terminal device is made to display a screen containing the aforementioned text information. Three-dimensional image management server.

16. An information processing method performed by an information processing system including a three-dimensional image management server for managing three-dimensional image information of an object, and a terminal device capable of communicating with the three-dimensional image management server, The aforementioned three-dimensional image management server is A first model that has learned the correspondence between the three-dimensional image information of the object, a predetermined region image in the captured image obtained by photographing the object with a photography device, and the input information input to the terminal device, The system includes a second model that has learned the correspondence between the three-dimensional image information of the object and the input information input to the terminal device, The terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model, or The terminal device generates text information relating to the object using the three-dimensional image information of the object selected and the second model. The steps include: causing the terminal device to display a screen containing the aforementioned text information; An information processing method that performs this task.

17. A 3D image management server manages the 3D image information of objects that can communicate with terminal devices. A first model that has learned the correspondence between the three-dimensional image information of the object, a predetermined region image in the captured image obtained by photographing the object with a photography device, and the input information input to the terminal device, A second model that has learned the correspondence between the three-dimensional image information of the object and the input information input to the terminal device, The terminal device generates text information about the object using the three-dimensional image information of the object selected, the predetermined region image, and the first model, or A text information generation unit generates text information relating to the object using the three-dimensional image information of the object selected by the terminal device and the second model. To make it function as, The terminal device is made to display a screen containing the aforementioned text information. program.

Images