Information processing systems, servers, information processing methods, programs

Abstract

The server that has established a session can perform processing based on three-dimensional image information displayed on any of the multiple display screens, regardless of which of the multiple display screens is being accessed. [Solution] The present invention is an information processing system 100 in which a terminal device displays a first display screen that includes information about an article received from the first server when the terminal device has established a session with the first server, and three-dimensional image information of the article received from the second server when the terminal device has established a session with both the first and second servers, and displays a second display screen that includes three-dimensional image information of the article received from the second server when the terminal device has established a session with the second server, and the second server performs processing based on the input information and three-dimensional image information of the article received for the first display screen, or processing based on the input information and three-dimensional image information received for the second display screen.

Description

Technical Field

[0001] The present invention relates to an information processing system, a server, an information processing method, and a program.

Background Art

[0002] There are cases where a first server and a second server each manage related information. In such a case, a technique is known in which a terminal device displays information managed by the first server and information managed by the second server (see, for example, Patent Document 1).

[0003] Patent Document 1 discloses a system that displays related information of a property transmitted from a link information management system and a full-spherical image of the property transmitted from an image management system on a communication terminal.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention provides a technique for executing processing based on three-dimensional image information that is displayed on any of a plurality of display screens and on all of the plurality of display screens from any of the servers establishing a session.

Means for Solving the Problems

[0005] The present invention is an information processing system including a first server that manages a two-dimensional image of an article and information about the article, a second server that manages three-dimensional image information of the article, and a terminal device capable of communicating with the first server and the second server. The terminal device includes a display control unit that displays a first display screen including information about the article received from the first server when a session is established with the first server, and three-dimensional image information of the article received from the second server when a session is established with both the first and second servers, and displays a second display screen including three-dimensional image information of the article received from the second server when a session is established with the second server. The second server processes the input information received by the input receiving unit of the terminal device and the three-dimensional image information of the article on the first display screen, or The device has a processing unit that performs processing on the second display screen based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information. [Effects of the Invention]

[0006] This technology allows for the execution of processing based on three-dimensional image information displayed on any of multiple display screens, regardless of which of the multiple display screens the server establishing the session is accessing. [Brief explanation of the drawing]

[0007] [Figure 1] This is an overall configuration diagram of an example of an information processing system. [Figure 2] This is a hardware configuration diagram of an example of an image management server, property information management server, or terminal device. [Figure 3] This is an example functional configuration diagram illustrating the functions of the image management server 40, the property information management server 20, and the terminal device 10 in an information processing system. [Figure 4] This is a conceptual diagram showing an example of an image information management table according to this embodiment. [Figure 5] This is a conceptual diagram showing an example of a property management information management table according to this embodiment. [Figure 6] This is an explanatory diagram displayed by the terminal device during text information generation processing and model update processing. [Figure 7] This is an explanatory diagram displayed by the terminal device during text information generation processing and model update processing. [Figure 8] This sequence diagram shows an example of a measurement process in which a user requests an image management server to measure the distance between two points on an object or property. [Figure 9] This diagram shows an example of a property management screen. [Figure 10] This diagram shows an example of a property management screen. [Figure 11] This is a diagram showing an example of a property display screen. [Figure 12] This figure shows an example of a property display screen where measurement results are shown. [Figure 13] This sequence diagram shows an example of a measurement process in which a user requests an image management server to measure the distance between two points on an object or property. [Figure 14] This figure shows an example of a property display screen that includes three-dimensional image information of the property. [Figure 15] This figure shows an example of a property display screen where measurement results are shown. [Figure 16] This is a sequence diagram showing an example of a model update process. [Figure 17] This is an example flowchart illustrating how the decision-making unit determines whether to update the first tacit knowledge model or the second tacit knowledge model. [Figure 18] This figure shows an example property display screen that includes property management information and 3D image information. [Figure 19] This is a sequence diagram showing an example of the text information generation process. [Figure 20] This figure shows an example of a property display screen during the inference phase. [Figure 21] This diagram shows an example of a text image screen that includes text information. [Figure 22] This is a sequence diagram showing an example of a model update process. [Figure 23] This is a diagram showing an example of a property display screen. [Figure 24] This is a sequence diagram showing an example of the text information generation process. [Figure 25] It is a diagram showing an example of a property display screen in the inference phase. [Figure 26] It is a diagram showing an example of a text image screen including text information. [Figure 27] It is a sequence diagram showing an example of a process in which an image management server communicates with a property information management server to update a model. [Figure 28] It is a sequence diagram explaining an example of a process in which an image management server executes a simulation. [Figure 29] It is a diagram showing an example of a property display screen. [Figure 30] It is a diagram showing an example of a simulation result screen. [Figure 31] It is a functional configuration diagram explaining an example of the functions of an image management server, a property information management server, and a terminal device in an information processing system. [Figure 32] It is a sequence diagram showing an example of a generation process of text information and image information. [Figure 33] It is a diagram showing the generated image information displayed on the text image screen.

Embodiments for Carrying Out the Invention

[0008] Hereinafter, as an example of an embodiment for carrying out the present invention, an information processing system and an information processing method performed by the information processing system will be described with reference to the drawings.

[0009] <Supplement Regarding Tacit Knowledge> In industries such as civil engineering and architecture, BIM / CIM is being promoted for the purpose of addressing issues such as the declining birthrate and aging population, and improving labor productivity.

[0010] BIM stands for Building Information Modeling, and it is a solution for utilizing information in all stages of construction, from design and construction to maintenance and management, by using a database of buildings that adds attribute data such as cost, finishes, and management information to a three-dimensional digital model of a building created on a computer (hereinafter referred to as the 3D model).

[0011] CIM stands for Construction Information Modeling and is a solution for the civil engineering sector (including infrastructure such as roads, power, gas, and water) that was proposed following the example of BIM, which was being developed in the building sector. Similar to BIM, CIM aims to improve the efficiency and sophistication of the entire construction production system by sharing information among stakeholders, primarily using 3D models.

[0012] A crucial aspect of promoting BIM and CIM implementation is how to effectively utilize the BIM and CIM systems that have been constructed.

[0013] Specifically, 3D models restored using BIM and CIM can be used not only for design and construction purposes, but also for other tasks such as maintenance and site surveys. In other words, they can be used not only as design drawings, but also for other purposes such as recording information in the 3D model and sharing it with others.

[0014] Furthermore, since the work performed on the 3D model can be recorded as a log, if tacit knowledge can be extracted from this data, it can be effectively used for transferring skills from experienced personnel to younger ones. This is expected to lead to front-loading of operations and talent development.

[0015] Focusing on the transmission of tacit knowledge, the challenge lies not only in 3D models but also in 2D data (such as 3D spherical images and planar images), as mentioned above, in how to transmit tacit knowledge between different tasks and among users with varying levels of skill.

[0016] Specifically, because tacit knowledge is qualitative and difficult to quantify, even if an tacit knowledge model is generated from tacit knowledge, it is difficult to secure user trust in the tacit knowledge model, making it difficult to promote its use. For example, if the field of expertise of the tacit knowledge model differs from that of the user, no matter how excellent the tacit knowledge model is, it will have no value to the user. Similarly, if the knowledge level of the tacit knowledge model is lower than the user's knowledge level, it will have no value to the user.

[0017] However, it is also true that tacit knowledge models can give users new perspectives and insights, and by using tacit knowledge models, even inexperienced users have great potential to acquire know-how and skills and apply them to their work.

[0018] Furthermore, there are cases where the first server and the second server manage related information. For example, the first server might hold property management information, while the second server manages the property's three-dimensional image information. In this case, if a user logs in to the first server from a terminal device, they can view the property management information. On the other hand, if a user logs in directly to the second server from a terminal device, even if the second server can perform processing according to the property management information, it may not be appropriate to provide the processing results to the user.

[0019] For example, if a user logs in directly to the second server, it is unclear whether the user has the authority to log in to the first server. Therefore, if the second server has a first tacit knowledge model obtained by learning inspection information of items, and the use of the first tacit knowledge model is permitted when a user logs in directly to the second server, it would mean providing processing to a user who does not have the authority to use it.

[0020] Therefore, in this embodiment, we will describe an information processing system that performs processing in response to a user logging into a first server and processing in response to a user logging into a second server.

[0021] <About Terminology> A user is someone who uses the text information (or other content such as images) generated by the tacit knowledge model. A data provider is someone who provides the data (such as audio information, text information, operation information, images, 3D data, etc.) that the tacit knowledge model uses for training.

[0022] Tacit knowledge is knowledge based on an individual's experience, intuition, etc. An tacit knowledge model is a model that learns tacit knowledge and outputs answers based on that learned tacit knowledge in response to questions. A model is a mechanism or artificial intelligence (AI) that learns the correspondence between input data and output data and outputs output data for given input data. Note that the output data does not depend on whether or not there is training data.

[0023] A property can be any space where items can be placed, such as a facility or a room within a facility. Items are things that are placed in the property. The types of items placed will vary depending on the function of the facility.

[0024] For example, properties include real estate, factories, construction sites, research facilities, medical facilities, farmland, warehouses, and equipment requiring maintenance. Goods include furniture, materials, equipment, heavy machinery, tools, instruments, materials, cultured products, and food.

[0025] Information regarding an item preferably includes information that can be considered tacit knowledge, such as the item's name or the name of the property where the item is located. For example, information regarding an item may include information known only to a specific user. In this embodiment, information regarding an item is described using the term "inspection information." Other examples of information regarding an item may include the item's instruction manual, daily reports of work performed on the item, an item's quotation, or an item's drawing. Inspection may also include investigation, testing, verification, review, checking, etc.

[0026] Three-dimensional image information of an object refers to the image of the object displayed on a screen. While three-dimensional image information of an object is a two-dimensional image projected from the three-dimensional model shape of the object, the user can change their viewpoint. Furthermore, three-dimensional image information of an object may also be a 360-degree spherical image.

[0027] The first process is updating the first tacit knowledge model or generating text information using inspection information, and the second process is updating the second tacit knowledge model or generating text information without using inspection information (using input information). The third process is a function that only the image management server 40 has, such as simulation.

[0028] [First Embodiment] <Example System Configuration> Figure 1 is an overall configuration diagram of an information processing system 100 according to an embodiment of the present invention. The information processing system 100 of this embodiment includes a terminal device 10, an image management server 40, and a property information management server 20, which are examples of input / output devices. However, the terminal device 10 may be connected to the image management server 40 or the property information management server 20 as needed, and the information processing system 100 does not necessarily have to include it.

[0029] The image management server 40 (an example of a second server) is one or more information processing devices that can communicate with the terminal device 10 via a communication network N. The image management server 40 manages three-dimensional image information of objects and also has an implicit knowledge model and a large-scale language model. The image management server 40 uses these to return text information containing implicit knowledge to the user. The image management server 40 may be a web server that returns processing results to the terminal device 10 in response to requests from the terminal device 10. A server is a computer or software that performs the function of providing information and processing results in response to requests from a client.

[0030] The image management server 40 may support cloud computing. Cloud computing refers to a usage model in which network resources are utilized without being aware of specific hardware resources. Cloud computing can take the form of SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service), and any of these forms is acceptable. Therefore, the image management server 40 does not need to be housed in a single enclosure or provided as a single unit of equipment. The image management server 40 may have its functions distributed among multiple information processing devices, or multiple information processing devices may each have all the functions, and the information processing device may be switched by load balancing or other means.

[0031] Furthermore, instead of the image management server 40 having both an implicit knowledge model and a large-scale language model, the image management server 40 may call an API (Application Programming Interface) exposed by an external system and utilize at least one of the implicit knowledge model or the large-scale language model.

[0032] The property information management server 20 (an example of the first server) is one or more information processing devices that can communicate with the terminal device 10 via the communication network N. The property information management server 20 manages property management information. Property management information is, for example, a string of characters such as text, and the property information management server 20 does not possess three-dimensional image information, or if it does, it is merely a photograph or something similar that is different from the images managed by the image management server 40. The property information management server 20 is a server for users to update property management information as needed to manage the progress of construction work on the property and the placement of items.

[0033] The property information management server 20 may be a web server that returns processing results to the terminal device 10 in response to requests from the terminal device 10. The property information management server 20 can communicate with the image management server 40 via the communication network N. The property information management server 20 may also support either cloud computing or on-premises deployment.

[0034] Terminal device 10 is a general-purpose information processing terminal used by users of the information processing system 100. Terminal device 10 runs a web browser and a dedicated native application on the image management server 40 or property information management server 20. When terminal device 10 runs a web browser, terminal device 10 and the image management server 40 or property information management server 20 run a web application. A web application is an application that operates through the cooperation of a program written in a programming language (e.g., JavaScript®) that runs on a web browser and a program on the web server (image management server 40). When a web application is executed, the processing of this embodiment may be performed by the image management server 40 or property information management server 20, or by terminal device 10 that receives the web application.

[0035] An application that cannot be executed unless it is installed on the terminal device 10 is called a native application. In this embodiment as well, the application executed on the terminal device 10 may be either a web application or a native application. In this case as well, the processing in this embodiment may be performed by the image management server 40 or by the terminal device 10 that is running the native application.

[0036] Terminal device 10 can be, for example, a PC (Personal Computer), a smartphone, a PDA (Personal Digital Assistant), or a tablet device. Alternatively, terminal device 10 can be any device capable of running a web browser or native applications. Terminal device 10 may also be an electronic whiteboard, a television receiver, a glasses device, or a wearable device. Furthermore, there may be multiple terminal devices 10.

[0037] The terminal device 10 can communicate with the image management server 40 and the property information management server 20 via the communication network N. The communication network N is constructed using the Internet, LAN (Local Area Network), provider services, etc. The communication network N may include not only wired communication but also mobile communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), and wireless LAN networks. Furthermore, the terminal device 10 can communicate using short-range communication technologies such as Bluetooth (registered trademark) and NFC (registered trademark, Near Field Communication).

[0038] In Figure 1, the image management server 40, the property information management server 20, and the terminal device 10 communicate via the communication network N. However, users may also operate the image management server 40 or the property information management server 20 directly from a console.

[0039] <Example Hardware Configuration> Figure 2 is a hardware configuration diagram of the image management server 40, property information management server 20, or terminal device 10 according to this embodiment. The hardware configurations of the image management server 40 or property information management server 20 are indicated by reference numerals in the 400 series. The hardware configurations of the terminal device 10 are indicated by reference numerals in the 100 series.

[0040] The hardware configuration of the terminal device 10 will be described below, but the hardware configuration of the image management server 40 or the property information management server 20 is the same and will therefore not be described.

[0041] The terminal device 10 is built using a computer and, as shown in Figure 2, includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HD (Hard Disk) 104, HDD (Hard Disk Drive) controller 105, display I / F 106, and communication I / F 107.

[0042] Of these components, the CPU 101 controls the overall operation of the terminal device 10. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101.

[0043] HD104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to HD10 according to the control of the CPU 101.

[0044] The display I / F 106 is a circuit that displays images on the display 106a. The display 106a is a type of display unit such as a liquid crystal or organic EL (electroluminescence) that displays various information such as cursors, menus, windows, characters, or images. The communication I / F 107 is an interface used for communication with other devices.

[0045] If the terminal device 10 is a glass device, the terminal device 10 may use a circuit that displays an image on a lens or the like, which is a transmissive reflective material, instead of the display I / F 106.

[0046] Communication I / F107 is, for example, a NIC (Network Interface Card) that supports TCP (Transmission Control Protocol) / IP (Internet Protocol).

[0047] Furthermore, the terminal device 10 is equipped with a sensor I / F 108, an audio input / output I / F 109, an input I / F 110, a media I / F 111, and a DVD-RW (Digital Versatile Disk Rewritable) drive 112.

[0048] The sensor interface 108 is an interface for receiving detection information from various sensors. The sound input / output interface 109 is a circuit that processes the input and output of sound signals between the speaker 109a and the microphone 109b according to the control of the CPU 101. The input interface 110 is an interface for connecting a predetermined input means to the terminal device 10.

[0049] Keyboard 110a is a type of input device equipped with multiple keys for inputting characters, numbers, various instructions, etc. Mouse 110b is a type of input device used for selecting and executing various instructions, selecting processing targets, moving the cursor, and operating on the display screen, etc.

[0050] The media interface 111 controls the reading or writing (storage) of data to or from a recording medium 111a such as flash memory. The DVD-RW drive 112 controls the reading or writing of various types of data to or from a DVD-RW 112a, which is an example of a removable recording medium. Note that the DVD-RW may be a DVD-R or the like. The DVD-RW drive 112 may also be a Blu-ray drive that controls the reading or writing of various types of data to or from a Blu-ray Disc (registered trademark).

[0051] Furthermore, the terminal device 10 is equipped with a bus line 113. The bus line 113 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 101.

[0052] Furthermore, recording media such as HDs and CD-ROMs on which the above programs are stored can be provided domestically or internationally as program products. The terminal device 10 realizes the information processing method according to the present invention by executing, for example, the program according to the present invention.

[0053] <About the features> Figure 3 is an example of a functional configuration diagram illustrating the functions of the image management server 40, the property information management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment.

[0054] <<Terminal device>> As shown in Figure 3, the terminal device 10 includes a transmitting / receiving unit 11, an input receiving unit 12, a display control unit 13, an audio control unit 14, a conversion unit 15, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 101 that follow a program deployed from the HD 104 onto the RAM 103. The terminal device 10 also has a storage unit 1000 constructed from at least one of the RAM 103 and HD 104 shown in Figure 2.

[0055] The transmitting / receiving unit 11 is an example of a transmission means and is implemented by commands from the CPU 101 shown in Figure 2 and the communication I / F 107, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0056] The input receiving unit 12 is an example of an input receiving means and is mainly implemented by commands from the CPU 101 shown in Figure 2, as well as the input I / F 110 and the sound input / output I / F 109, and accepts various inputs from the user using the microphone 109b, keyboard 110a and mouse 110b.

[0057] The display control unit 13 is an example of a display control means and output means, and is implemented by commands from the CPU 101 shown in Figure 2 and the display I / F 106, and causes various images and screens to be displayed on the display 106a, which is an example of a display unit. If the terminal device 10 is a glasses device, the display control unit 13 causes a virtual image to be displayed on a transmissive reflective member such as a lens instead of the display I / F 106.

[0058] The audio control unit 14 is an example of an audio control means and output means, and is implemented by commands from the CPU 101 shown in Figure 2 and the audio input / output I / F 109, causing the speaker 109a, which is an example of an audio playback unit, to play sound.

[0059] The conversion unit 15 is an example of a processing means, and is implemented by instructions from the CPU 101 shown in Figure 2. It performs processing to convert text information into audio information and processing to convert audio information into text information.

[0060] The storage / reading unit 19 is an example of a storage control means and is executed by instructions from the CPU 101 shown in Figure 2, as well as by the HD 104, media I / F 111, and DVD-RW drive 112. It performs processing such as storing various data in the storage unit 1000, recording media 111a, and DVD-RW 112a, and reading various data from the storage unit 1000, recording media 111a, and DVD-RW 112a.

[0061] <Image management server functional configuration> The image management server 40 includes a transmission / reception unit 41, a screen generation unit 42, a judgment unit 43, a specification unit 44, a text information generation unit 45, an update unit 46, a processing unit 47, a determination unit 48, a measurement unit 51, a simulation unit 52, and a storage / reading unit 49. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 following a program deployed from the HD 404 onto the RAM 403. The image management server 40 also has a storage unit 4000 constructed from the HD 404 shown in Figure 2. The storage unit 4000 is an example of a storage means.

[0062] In Figure 3, all functions are installed in a single image management server 40, but the image management server 40 may also be configured to distribute each function across multiple computers.

[0063] The transmitting / receiving unit 41 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0064] The screen generation unit 42 is an example of a screen generation means and is implemented by instructions from the CPU 401 shown in Figure 2, generating various screens. When the terminal device 10 runs a web application, screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc. Therefore, screen information may also be referred to as a web application. When the terminal device 10 runs a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0065] The decision unit 43 is an example of a decision-making mechanism, and is implemented by instructions from the CPU 401 shown in Figure 2, and performs various decisions described later.

[0066] The identification unit 44 is an example of an identification means, and is implemented by instructions from the CPU 401 shown in Figure 2, and identifies the target image.

[0067] The text information generation unit 45 is an example of a text information generation means, and is implemented by instructions from the CPU 401 shown in Figure 2. It acquires tacit knowledge comments based on the first tacit knowledge model 4004A or the second tacit knowledge model 4004B, which will be described later, or generates text information based on the large-scale language model 4005.

[0068] The update unit 46 is an example of an update means, which is implemented by instructions from the CPU 401 shown in Figure 2, and updates the first tacit knowledge model 4004A or the second tacit knowledge model 4004B.

[0069] The processing unit 47 is implemented by instructions from the CPU 401 shown in Figure 2, and performs processing to associate three-dimensional image information and captured images with text information, which is an example of generated information, according to the processing requested by the user. The generated information is generated based on the three-dimensional image information and captured images and inspection information or inspection information. Processing based on the inspection information and image information includes processing to display both on one screen, processing to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B, processing to generate text information based on the first tacit knowledge model 4004A or the second tacit knowledge model 4004B, etc. The processing unit 47 requests processing from the screen generation unit 42 or the text information generation unit 45, etc., depending on the content of the processing.

[0070] The decision unit 48 is implemented by instructions from the CPU 401 shown in Figure 2 and determines whether to use the first tacit knowledge model 4004A (an example of the first model) or the second tacit knowledge model 4004B (an example of the second model). The decision unit 48 determines to use the first tacit knowledge model 4004A if the user logs in to the image management server 40 via the property information management server 20. The decision unit 48 determines to use the second tacit knowledge model 4004B if the user logs in to the image management server 40 directly.

[0071] The measurement unit 51 is implemented by instructions from the CPU 401 shown in Figure 2, and measures the distance between two points specified by the user.

[0072] The simulation unit 52 is implemented by instructions from the CPU 401 shown in Figure 2, and executes pre-configured simulations on an object or item. The simulations include various types, such as predicting airflow within an object, predicting the size of items within an object (e.g., whether they can be brought in or installed), counting the number of items, or calculating the volume of a space.

[0073] The storage / reading unit 49 is an example of a storage control means and is executed by instructions from the CPU 401 shown in Figure 2, as well as by the HD 404, media I / F 411, and DVD-RW drive 412. It performs processing such as storing various data in the storage unit 4000, recording media 411a, and DVD-RW 412a, and reading various data from the storage unit 4000, recording media 411a, and DVD-RW 412a. The storage unit 4000, recording media 411a, and DVD-RW 412a are examples of storage means.

[0074] The memory unit 4000 contains an image information management DB 4001, a model shape management DB 4002, a caption model 4003, a first tacit knowledge model 4004A, a second tacit knowledge model 4004B, and a large-scale language model 4005.

[0075] Image information management DB 4001 manages three-dimensional image information related to the property. This three-dimensional image information is information that visually represents the items (also called models) placed on the property. Model shape management DB 4002 manages three-dimensional model shape information of the items placed on the property. Image management server 40 can generate three-dimensional image information related to the property from the three-dimensional model shape information. Three-dimensional model shape information is information that renders an item in three dimensions, such as a three-dimensional model of the item or a three-dimensional point cloud. Three-dimensional model shape information may be composed of polygons or CAD. In addition, it is preferable that image information management DB 4001 or model shape management DB 4002 store wide-field images such as 360-degree spherical images of the property.

[0076] Caption model 4003 is a model that is generated by performing a training process using image and caption comment combinations as training data, and enables a computer to function to output caption comments based on images. Caption comments are explicit knowledge and are used as a term in contrast to tacit knowledge. Here, caption comments are text data, such as comments expressed in audio or text that describe an image. Caption comments relating to objects or articles are associated with the identification information of those objects or articles.

[0077] The first tacit knowledge model 4004A is generated by performing a learning process using the correspondence between three-dimensional image information of an item and inspection information as training data, and is a model that causes a computer to function to output tacit knowledge comments based on the image. Here, the tacit knowledge comment is text data, and is a comment expressed in voice or text, excluding caption comments, that is, a comment relating to content not expressed in the image. The first tacit knowledge model 4004A may also be generated by performing a learning process using the correspondence between three-dimensional image information of an item, inspection information and comments as training data.

[0078] The second tacit knowledge model 4004B does not use inspection information for learning. In other words, the second tacit knowledge model 4004B is generated by performing a learning process using the correspondence between three-dimensional image information of an item and comments as learning data, and it is a model that makes the computer function to output tacit knowledge comments based on the image.

[0079] The large-scale language model 4005 is a computer language model that is generated by performing a training process using a vast amount of unlabeled text as training data and consists of an artificial neural network with many parameters. The large-scale language model 4005 can capture much of the syntax and meaning of human language by being sufficiently trained with context-learning techniques such as next sentence prediction, which understands context by determining whether sentence 1 and sentence 2 are consecutive, and the masked language model, which understands context by masking words in a sentence and predicting the masked words from the words before and after them.

[0080] <<Image Information Management Table>> Figure 4 is a conceptual diagram showing an example of an image information management table according to this embodiment. The storage unit 4000 has an image information management DB 4001 constructed, which is composed of the image information management table shown in Figure 4.

[0081] This image information management table manages model identification information, location information, item information, inspection information, 3D image information, and captured images, linked to the item identification information. An item can be any space where items can be placed, such as a facility or a room within a facility. The types of items placed will vary depending on the function of the facility.

[0082] Property identification information is identification information that uniquely identifies a property.

[0083] Model identification information is identification information that identifies items placed on a property. Each item is associated with the three-dimensional model shape in the model shape management DB4002 using the model identification information. This model identification information is the same as the model identification information in the property information management server 20, but it may also be associated with the model identification information in the property information management server 20 using a conversion table.

[0084] • Item information refers to the names of the items placed on the property. In Figure 4, item names are shown assuming a construction site or building site, but item names will vary depending on the property. Also, in Figure 4, properties and items are associated, but items can be registered independently of properties.

[0085] • Location information is information that indicates the position of an object in a virtual space within the generated property using three-dimensional coordinates (XYZ). XYZ can be the centroid of the object, or the coordinates of the vertices of the circumscribing cube.

[0086] Inspection information is information about inspections performed by users on items within a property. Inspection information varies depending on the item and the purpose of the inspection, but may include, for example, whether there are any abnormalities in the item, dimensions, inspection date and time, and inspector. Inspection information is obtained from the property information management server 20. The property information management server 20 holds master inspection information, and the image management server 40 obtains and stores inspection information from the property information management server 20. This is because item inspections are performed on the property information management server 20. For this reason, inspection information in the image information management table may only exist for some items, or even if inspection information exists, it may not be the most up-to-date inspection information.

[0087] In addition, the inspection information may also be photographic information of the inspected items. In this case, the update unit 46 analyzes the photographic information to extract features such as the items shown and the condition of any damage, and treats them as string-based inspection information.

[0088] In addition to inspection information, instruction manuals, daily reports, quotations, drawings, etc., may also be registered in the image information management table.

[0089] Three-dimensional image information refers to information that visually represents an object, such as three-dimensional model shape information or a 360-degree image. In Figure 4, three-dimensional image information is placed in the image information management table, but the three-dimensional image information may also be stored in the model shape management DB4002.

[0090] Images 1 and 2 are two-dimensional images obtained from the property information management server 20 along with inspection information. The property information management server 20 holds the master images. The images are, for example, two-dimensional images of items taken by inspectors during inspection. Therefore, if inspection information is stored in the image information management DB 4001, the images are often also stored there. There are multiple images, such as Images 1 and 2, because items may be photographed each time an inspection is performed. There may be more than three images. However, the most recent image may overwrite the previous one. Images 1 and 2 have a date and time of capture.

[0091] <Functional Configuration of Property Information Management Server> Let's return to Figure 3 for explanation. The property information management server 20 has a transmitting / receiving unit 21, a screen generation unit 22, and a storage / reading unit 29. Each of these units is a function or means of functioning that is realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 that follow a program deployed from the HD 404 onto the RAM 403. The property information management server 20 also has a storage unit 2000 constructed from the HD 404 shown in Figure 2. The storage unit 2000 is an example of a storage means.

[0092] In Figure 3, all functions are installed on a single property information management server 20, but the property information management server 20 may also be configured to distribute each function across multiple computers.

[0093] The transmitting / receiving unit 21 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0094] The screen generation unit 22 is an example of a screen generation means and is implemented by instructions from the CPU 401 shown in Figure 2, generating various screens. When the terminal device 10 runs a web application, screen information is created using HTML, XML, CSS, and JavaScript (registered trademark), etc. Therefore, screen information may also be referred to as a web application. When the terminal device 10 runs a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0095] The storage / reading unit 29 is an example of a storage control means and is executed by instructions from the CPU 401 shown in Figure 2, as well as by the HD 404, media I / F 411, and DVD-RW drive 412. It performs processing such as storing various data in the storage unit 2000, recording media 411a, and DVD-RW 412a, and reading various data from the storage unit 2000, recording media 411a, and DVD-RW 412a. The storage unit 2000, recording media 411a, and DVD-RW 412a are examples of storage means.

[0096] <<Property Management Information Management Table>> Figure 5 is a conceptual diagram showing an example of a property management information management table according to this embodiment. The storage unit 2000 has a property information management DB 2001 constructed, which is composed of property management information management tables as shown in Figure 5.

[0097] This property management information management table manages model identification information, location information, item information, inspection information, and one or more photographed images, associated with the property identification information. These are sometimes referred to as property management information. The contents of these are the same as those in the image information management table. In other words, the property management information management table does not store three-dimensional image information. However, inspection information and photographed images are recorded for all items that have been inspected at least once. Also, the inspection information and photographed images in the property information management server 20 are inspection information directly registered by the user, so they serve as the master (source) for the inspection information in the image management server 40. Photographed images may be photographs of multiple items, or they may be images other than photographs, such as layout diagrams.

[0098] Alternatively, instead of the property information management DB2001 consisting of the property management information management table described above, a property information management DB2001 consisting of a property management information management table that manages location information, item information, inspection information, and captured images, without associating them with property identification information, may be used.

[0099] <An example of model updating and text information generation> Refer to Figures 6 and 7 to explain how to update the model and generate text information. In Figures 6 and 7, check information is not used for updating the model and generating text information, but similar learning can be achieved by replacing or adding check information to dialogue Q1, etc. below.

[0100] Figure 6 is an explanatory diagram of what the terminal device 10 displays during text information generation processing and model update processing. Figure 6(a) is an explanatory diagram of the model update processing. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes the target image 1100 and text 1200.

[0101] The input receiving unit 12 of the terminal device 10 receives audio information from the microphone 109b, which represents the dialogue Q1, A1, Q2, and A2 between data providers M1 and M2, as input information to be entered by the data provider into the displayed screen 900. It is preferable that data providers M1 and M2 have extensive knowledge, including tacit knowledge, about their work. Through such a dialogue between data providers M1 and M2, the tacit knowledge model is updated, allowing the user to obtain useful tacit knowledge comments.

[0102] The identification unit 44 identifies the target image 1100, which is the portion of the display screen 900 excluding the text 1200.

[0103] The judgment unit 43 then uses the target image 1100 to determine the degree of relevance between the caption comments obtained from the caption model 4003 and the dialogues Q1, A1, Q2, and A2.

[0104] The update unit 46 updates the tacit knowledge model by using comments from the dialogues Q1, A1, Q2, and A2 that are judged to have a low degree of relevance as tacit knowledge comments, along with the target image 1100, etc., as training data, and updates the caption model 4003 by using comments that are judged to have a high degree of relevance as caption comments, along with the target image 1100, as training data.

[0105] Therefore, the tacit knowledge model learns the correspondence between the target image 1100 and the dialogues Q1, A1, Q2, and A2. The target image 1100 has features extracted using several feature extraction models suitable for images, such as a CNN (Convolutional Neural Network). These features represent, for example, what is depicted and where, or the tasks being performed within the image. Thus, the tacit knowledge model can learn the correspondence between the image features and the dialogues Q1, A1, Q2, and A2.

[0106] Figure 6(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes an image 1110 and text 1210.

[0107] The input receiving unit 12 of the terminal device 10 receives audio information indicating questions Q11 and Q12 from user M3 via the microphone 109b as input information to be entered by the user on the displayed screen 900.

[0108] The identification unit 44 identifies the image 1110 that does not contain the text 1210 as the target image.

[0109] The text information generation unit 45 uses image 1110 to acquire tacit knowledge comments based on the tacit knowledge model. The tacit knowledge model extracts features from image 1110 and determines that the features of image 1110 in Figure 6(b) are similar to those of image 1110 at the time of update, thereby identifying dialogues Q1, A1, Q2, and A2 related to image 1110. Dialogues Q1, A1, Q2, and A2 become tacit knowledge comments.

[0110] Furthermore, the text information generation unit 45 uses tacit knowledge comments (dialogue Q1, A1, Q2, A2) and questions Q11, Q12, etc., to generate text information related to the answers A11 and A12 to questions Q11 and Q12, respectively, based on the large-scale language model 4005.

[0111] The display control unit 13 of the terminal device 10 displays the text information related to the answers A11 and A12 received from the image management server 40 on the display 106a.

[0112] Figure 7 is another explanatory diagram displayed by the terminal device 10 during the model update process and text information generation process according to this embodiment. Figure 7 illustrates a case in which no question text is used during model update and text information generation.

[0113] Figure 7(a) is an explanatory diagram of the model update process. Figure 7(a) shows an example in which the tacit knowledge model is updated with audio information and partial images from a single data provider, rather than a conversation between data providers.

[0114] The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes a first image 1100A and a second image 1100B.

[0115] The input receiving unit 12 of the terminal device 10 receives character information from the keyboard 110a, which represents comments C1 to C4 from data provider M4, as input information to be entered by the data provider in response to the displayed screen 900.

[0116] Furthermore, the input receiving unit 12 receives operation information from the mouse 110b, which indicates an operation by the data provider M4 to identify a partial image 1100B1 in the second image 1100B, as input information to be entered by the data provider M4 into the displayed screen 900.

[0117] The identification unit 44 may identify the partial image 1100B1 as the target image, or it may identify the first image 1100A or the second image 1100B as the target image.

[0118] The judgment unit 43 then uses the target image to determine the degree of relevance between the caption comments obtained from the caption model 4003 and comments C1 to C4.

[0119] The update unit 46 updates the tacit knowledge model by using comments C1 to C4 that are judged to have low relevance as tacit knowledge comments, along with partial image 1100B1, etc., as training data, and updates the caption model 4003 by using comments that are judged to have high relevance as caption comments, along with partial image 1100B1, as training data.

[0120] Therefore, the tacit knowledge model learns the correspondence between the partial image 1100B1 and the comments C1-C4. Features are extracted from the partial image 1100B1 using several feature extraction models suitable for images, such as CNNs. These features represent, for example, what is depicted and where, or the work being performed. Thus, the tacit knowledge model can learn the correspondence between the image features and the comments C1-C4.

[0121] Figure 7(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes the image 1110.

[0122] User M5 does not input anything to the displayed screen 900, and the input reception unit 12 does not accept any input information from the user to the displayed screen 900. The identification unit 44 identifies the entire displayed screen 900, which is image 1110, as the target image.

[0123] When user M5 performs an operation to identify a partial image 1100B1 on the display screen 900, the input receiving unit 12 receives operation information from the mouse 110b indicating the operation to identify the partial image as input information. In this case, the identification unit 44 identifies the partial image on the display screen 900 as the target image according to the operation information.

[0124] The text information generation unit 45 uses the partial image 1100B1 to acquire tacit knowledge comments based on the tacit knowledge model. The tacit knowledge model determines that the features of image 1110B1 in Figure 7(b) are similar to those of image 1110B1 at the time of update, and can identify comments C1 to C4 related to image 1110B1. The tacit knowledge model extracts comments C1 to C4 as tacit knowledge comments. The text information generation unit 45 generates text information related to comments C11 to C14 based on the large-scale language model 4005, using tacit knowledge comments and the like. The text information generation unit 45 may generate text information using preset standard questions if no question text is entered, rather than using a method that does not use any questions at all.

[0125] The display control unit 13 of the terminal device 10 displays the text information related to comments C11 to C14 received from the image management server 40 on the display 106a.

[0126] <Action or process> <<Login to the image management server via the property information management server>> First, we will explain the case where a user measures the distance between two points on a property's three-dimensional image data. The user may log in to the image management server 40 via the property information management server 20, or they may log in directly to the image management server 40. Regardless of how the user logs in, the measurement results will be the same.

[0127] Referring to Figure 8, we will explain the case where a user logs in to the image management server 40 via the property information management server 20. Figure 8 is a sequence diagram showing an example of a measurement process in which a user requests the image management server 40 to measure the distance between two points on an item or property.

[0128] S1: The user enters a login operation into terminal device 10. This login is for the property information management server 20. The input reception unit 12 of terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful. Upon successful login, a session is established between terminal device 10 and property information management server 20. Processing between terminal device 10 and property information management server 20 is performed while the session is established.

[0129] Users log in to the property information management server 20 first, and then log in to the image management server 40. However, they may also log in to the image management server 40 first, and then log in to the property information management server 20.

[0130] S2: Upon successful login, the transmitting / receiving unit 11 of the terminal device 10 sends a request for the property selection screen 200 to the property information management server 20.

[0131] S3: The transmitting / receiving unit 21 of the property information management server 20 receives a request for the property selection screen 200. The screen generation unit 22 generates the property selection screen 200, and the transmitting / receiving unit 21 transmits the screen information of the property selection screen 200 to the terminal device 10.

[0132] S4: The transmitting / receiving unit 11 of the terminal device 10 receives screen information from the property selection screen 200. The display control unit 13 displays the property selection screen 200 (see Figure 9). The user enters property identification information (e.g., V0001) on the displayed property selection screen 200. The input receiving unit 12 of the terminal device 10 receives the property identification information.

[0133] S5: The transmitting / receiving unit 11 of the terminal device 10 sends a request for property management information to the property information management server 20, specifying the property identification information.

[0134] S6: The transmitting / receiving unit 21 of the property information management server 20 receives a request for property management information, and the storage / reading unit 29 searches the property information management DB 2001 using the property identification information. The property management information is information about the property associated with the property identification information. The screen generation unit 22 of the property information management server 20 generates a property management screen 210 that displays the property management information, and the transmitting / receiving unit 21 transmits the screen information of the property management screen 210 to the terminal device 10.

[0135] Furthermore, the transmitting / receiving unit 21 sends an image request program to the terminal device 10 in response to a request for property management information, so that the terminal device 10 can acquire three-dimensional image information. The image request program is, for example, a web application. This web application is installed on the property information management server 20 by the operator of the image management server 40 with the permission of the operator of the property information management server 20. Alternatively, a URL containing the image request program may be sent to the terminal device 10. Since the web application acquires three-dimensional image information from the image management server 40, it has the function of connecting to the image management server 40 and requesting and displaying three-dimensional image information.

[0136] S7: The transmitting / receiving unit 11 of the terminal device 10 receives screen information and an image request program from the property management screen 210. The display control unit 13 displays the property management screen 210 (see Figure 10). This displays the property management information. The user requests three-dimensional image information of the property (by pressing the image acquisition button 213) based on the displayed property management information. The input receiving unit 12 of the terminal device 10 receives the request for three-dimensional image information of the property. Three-dimensional image information of the property is three-dimensional image information of items placed in the property, which has been generated as a virtual space. The items are composed of 3D model shape information. Since the property has already been identified, the request for three-dimensional image information of the property may be sent to the image management server 40 without any operation by the user.

[0137] Here, the property management screen 210 includes a first display area 214 that displays information about the item obtained from the property information management server 20, and a second display area 215 that displays three-dimensional image information of the item obtained from the image management server 40. However, at step S7, the property management information is displayed in the first display area 214, but nothing is displayed in the second display area 215.

[0138] S8: If the user is not logged in to the image management server 40, the user enters a login operation into the terminal device 10. This login operation is directed to the image management server 40. The input reception unit 12 of the terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful. The image management server 40 may also omit the user's login operation by using, for example, single sign-on. Upon successful login, a session is established between the terminal device 10 and the image management server 40. Processing between the terminal device 10 and the image management server 40 is performed while the session is established.

[0139] S9: The terminal device 10 executes an image request program to request three-dimensional image information. The transmitting / receiving unit 11 then sends a request for three-dimensional image information of the property selected by the user to the image management server 40, specifying the property identification information of the property. The transmitting / receiving unit 11 should also send the URL of the property information management server 20 to the image management server 40 so that the terminal device 10 can be redirected to the property information management server 20. Three-dimensional image information of a property is an image of an item placed in the property, which is a virtual space. Since the item is composed of 3D model shape information, the terminal device 10 projects the 3D model shape of the item onto a two-dimensional plane to generate a planar image. The user can view any item while changing their viewpoint. The transmitting / receiving unit 11 may also have the image management server 40 receive property management information obtained from the property information management server 20. The image request program receives property management information from a web application connected to the property information management server 20, for example, as a URL parameter.

[0140] S10: The transmitting / receiving unit 41 of the image management server 40 receives requests for three-dimensional image information of an object along with object identification information. The storage / reading unit 49 searches the image information management DB 4001 using the object identification information and obtains three-dimensional image information for each item. The processing unit 47 requests the screen generation unit 42 to generate a screen containing the three-dimensional image information of the object. The screen generation unit 42 generates a screen corresponding to the second display area 215 in which the three-dimensional image information of each item is arranged. The transmitting / receiving unit 41 transmits the screen information of the screen corresponding to the second display area 215 to the terminal device 10. The three-dimensional image information of each item included in the screen information is three-dimensional image information in which all items included in the object are placed on the object, and the user can arbitrarily change the viewpoint.

[0141] S11: The transmitting / receiving unit 11 of the terminal device 10 receives screen information for the screen corresponding to the second display area 215, and the display control unit 13 displays the property display screen 220 including the first display area 214 and the second display area 215 (see Figure 11). Three-dimensional image information of each item is displayed in the second display area 215, and a list of items placed in the property (generated from property management information) is displayed in the first display area 214, for example, so that the item list and the three-dimensional image information of the property are displayed on one screen. Next, the user specifies two points on the three-dimensional image information of the property using a mouse cursor 212 or the like to measure dimensions. The input receiving unit 12 of the terminal device 10 receives the coordinates of the two points on the item.

[0142] S12: When the user presses the measurement button 229, the transmitting / receiving unit 11 of the terminal device 10 sends a measurement request to the image management server 40, specifying the coordinates of two points.

[0143] S13: The transmitting / receiving unit 41 of the image management server 40 receives a measurement request. The measurement unit 51 searches the image information management DB 4001 based on the coordinates and identifies the item (model identification information). The measurement unit 51 also searches the model shape management DB 4002 using the model identification information and obtains the three-dimensional model shape information of the item. The measurement unit 51 measures the dimension between the coordinates of two points based on the three-dimensional model shape information.

[0144] S14: The transmitting / receiving unit 41 of the image management server 40 transmits the measurement result (dimension between the coordinates of two points) to the terminal device 10.

[0145] S15: The transmitting / receiving unit 11 of the terminal device 10 receives the measurement results, and the display control unit 13 displays the measurement results along with the three-dimensional image information of the property (see Figure 12).

[0146] <<Screen example>> Figure 9 shows an example of a property selection screen 200 for entering property identification information. The property selection screen 200 has a property identification information input field 201 and a search button 202. When the user enters the property identification information in the property identification information input field 201 and presses the search button 202, the list of room numbers shown in Figure 10 is displayed on the property management screen 210.

[0147] Figure 10 shows an example of a property management screen 210. The property management screen 210 includes a first display area 214 that displays information about items obtained from the property information management server 20, and a second display area 215 that displays three-dimensional image information of items obtained from the image management server 40. The first display area includes a list 211 of room numbers belonging to properties identified by property identification information. The list 211 of room numbers corresponds to property management information, and includes items contained in each property. Depending on the property, the room number may not be displayed, and the user may transition from the property selection screen 200 in Figure 9 to Figure 11 to display the three-dimensional image information of the property. The user selects the room number for which they want to display the three-dimensional image information using the mouse cursor 212. When the user presses the image acquisition button 213, the property display screen 220 is displayed.

[0148] Figure 11 shows a property display screen 220 containing property management information and three-dimensional image information. The property display screen 220 includes a first display area 214 and a second display area 215, and the first display area 214 of the property display screen 220 displays an item list 221. The items in this room are the ceiling, floor, walls, and door. The second display area 215 of the property display screen 220 displays three-dimensional image information 227 of these items. The user specifies two points to measure using the mouse cursor 212. In Figure 11, two points on the wall are specified. When the user presses the measurement button 229, the image management server 40 starts measuring the distance between the two points.

[0149] Furthermore, the property display screen 220 displays the area (floor area) 224 as information about the property. The area (floor area) 224 may be a measured value or may be included in the property management information.

[0150] Note that the screen shown in Figure 11, which displays property management information (a list of room numbers 211 and item names) and three-dimensional image information of the property 227, is an example of the first display screen.

[0151] Figure 12 shows the property display screen 220 with the measurement results displayed. As can be seen by comparing it with Figure 11, the dimension 228 between the two points is displayed as the measurement result.

[0152] In this way, when a user logs in to the image management server 40 via the property information management server 20, the terminal device 10 can display property management information and three-dimensional image information of the property on a single screen. The user can then perform tasks using the three-dimensional image information of the property, such as measuring the distance 228 between two points, while viewing the property management information.

[0153] <<Direct login to the property information management server>> Next, we will explain the case where a user logs directly into the image management server 40 and measures the distance between two points.

[0154] Figure 13 is a sequence diagram showing an example of a measurement process in which a user requests the image management server 40 to measure the distance between two points on an object or property.

[0155] S101: The user enters a login operation into terminal device 10. This login is for the image management server 40. The input reception unit 12 of terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful.

[0156] S102: Upon successful login, the transmitting / receiving unit 11 of the terminal device 10 sends a request for the property selection screen 200 to the image management server 40.

[0157] S103: The transmitting / receiving unit 41 of the image management server 40 receives a request for the property selection screen 200. The screen generation unit 42 generates the property selection screen 200, and the transmitting / receiving unit 41 transmits the screen information of the property selection screen 200 to the terminal device 10.

[0158] S104: The transmitting / receiving unit 11 of the terminal device 10 receives screen information from the property selection screen 200. The display control unit 13 displays the property selection screen 200 (see Figure 9). The user enters property identification information (e.g., V0001) on the displayed property selection screen 200. The input receiving unit 12 of the terminal device 10 receives the property identification information.

[0159] S105: The transmitting / receiving unit 11 of the terminal device 10 sends a request for three-dimensional image information of the property to the image management server 40, specifying the property identification information. Note that since the terminal device 10 is not logged into the property information management server 20, the property management screen 210 shown in Figure 10 is not displayed.

[0160] S106: The transmitting / receiving unit 41 of the image management server 40 receives a request, and the storage / reading unit 49 searches the image information management DB 4001 using the item identification information. The storage / reading unit 49 acquires three-dimensional image information for each item. The screen generation unit 42 generates an item display screen 320 in which the three-dimensional image information of each item is arranged. The transmitting / receiving unit 41 transmits the screen information of the item display screen 320 to the terminal device 10. The three-dimensional image information for each item is three-dimensional image information in which all items included in the item are placed in the item, and the user can arbitrarily change the viewpoint.

[0161] S107: The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the property display screen 320, and the display control unit 13 displays the property display screen 320 (see Figure 14). In the process shown in Figure 13, since the terminal device 10 is not logged into the property information management server 20, the property display screen 320 does not display the item list 221 (property management information) placed on the property. However, the screen generation unit 42 may use the image information management DB 4001 managed by the image management server 40 to display information equivalent to the item list. Next, the user specifies two points on the three-dimensional image information of the property using a mouse cursor or the like to measure dimensions. The input receiving unit 12 of the terminal device 10 receives the coordinates of the two points on the property.

[0162] S108: When the user presses the measurement button 229, the transmitting / receiving unit 11 of the terminal device 10 sends a measurement request to the image management server 40, specifying the coordinates of two points.

[0163] S109: The transmitting / receiving unit 41 of the image management server 40 receives a measurement request. The measurement unit 51 searches the image information management DB 4001 based on the coordinates and identifies the item (model identification information). The measurement unit 51 also searches the model shape management DB 4002 using the model identification information and obtains the three-dimensional model shape information of the item. The measurement unit 51 measures the dimension between the coordinates of two points based on the three-dimensional model shape information.

[0164] S110: The transmitting / receiving unit 41 of the image management server 40 transmits the measurement result (distance between two points) to the terminal device 10.

[0165] S111: The transmitting / receiving unit 11 of the terminal device 10 receives the measurement result, and the display control unit 13 displays the measurement result on the property display screen 320 (see Figure 15).

[0166] <<Screen example>> The property selection screen 200 can be the same as in Figure 9. Also, if the user logs in directly to the image management server 40, the property management screen 210 in Figure 10 will not be displayed.

[0167] Figure 14 shows the property display screen 320, which includes three-dimensional image information of the property. Compared to Figure 11, the item list 221 is not displayed. This is because the item list 221 is managed by the property information management server 20. The method for the user to specify the coordinates of two points is the same as in Figure 11.

[0168] Note that the screen shown in Figure 14, which displays the property's three-dimensional image information 227 without displaying property management information, is an example of a second display screen.

[0169] Figure 15 shows the property display screen 320 with the measurement results displayed. Compared to Figure 12, the item list 221 is not displayed. This is because the item list 221 is managed by the property information management server 20. Note that the measurement result, the dimension 228 between the two points, is the same as in Figure 12.

[0170] Thus, when a user logs in directly to the image management server 40, the terminal device 10 displays only the three-dimensional image information 227 of the property. Therefore, if a user does not have the authority to use the property information management server 20, the image management server 40 can be restricted from providing the user with the information managed by the property information management server 20.

[0171] <Main effects> According to this embodiment, the image management server 40 processes information in response to a user logging into the first server and processes information in response to a user logging into the second server. In other words, the information displayed on the terminal device 10's screen can be changed depending on whether the user logs into the image management server 40 directly or via the property information management server 20. If the user does not have the authority to use the property information management server 20, the image management server 40 can be restricted from providing information managed by the property information management server 20. Furthermore, the image management server 40 can provide the same measurement results whether the user logs into the image management server 40 directly or via the property information management server 20.

[0172] [Second Embodiment] In this embodiment, we will describe an information processing system 100 that uses a first tacit knowledge model 4004A and a second tacit knowledge model 4004B depending on whether the user logged in to the image management server 40 via the property information management server 20 or logged in to the image management server 40 directly.

[0173] In this embodiment, the hardware configuration diagram shown in Figure 2 and the functional block diagram shown in Figure 3, as described in the above embodiment, can be used as references.

[0174] <Action or process> <<Login to the image management server via the property information management server>> First, we will explain the case where a user logs in to the image management server 40 via the property information management server 20. We will also explain the model update process in which the first tacit knowledge model 4004A learns data.

[0175] ●Learning Phase (Model Update) Figure 16 is a sequence diagram showing an example of the model update process. The processing in steps S21 to S30 can be the same as in Figure 8.

[0176] S31: The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215, and the display control unit 13 displays the property display screen 330 including the first display area 214 and the second display area 215 (see Figure 18). For example, a list of items placed in the property and three-dimensional image information of the property are displayed on one screen. Next, the user identifies an arbitrary item from the three-dimensional image information of the property. Once the user identifies an item, they can request the generation of inspection information for the item or text information using an implicit knowledge model. The input receiving unit 12 of the terminal device 10 accepts the operation to identify an item. The item may be identified, for example, by the coordinates clicked by the user, or model identification information may be identified by these coordinates.

[0177] The user inputs comments (text information, audio) about the item into the terminal device 10, as explained in Figures 6 and 7. These comments are sometimes referred to as input information. These comments may constitute tacit knowledge comments. Furthermore, the comments may include caption comments describing the item.

[0178] S32: When the user presses the information update button 226, the transmitting / receiving unit 11 of the terminal device 10 transmits to the image management server 40 that the information update button 226 has been pressed, along with information identifying the item and the input information.

[0179] S33: The transmitting / receiving unit 41 of the image management server 40 receives information that the information update button 226 has been pressed, as well as information identifying the item and input information. The storage / reading unit 49 identifies the model identification information of the item with the closest location information from the image information management DB 4001 based on the information identifying the item (in this case, location information). The transmitting / receiving unit 41 transmits the model identification information to the terminal device 10. Note that if the information identifying the item in step S32 is model identification information, searching the image information management DB 4001 is unnecessary. The image management server 40 transmits the model identification information in order to obtain the latest inspection information and captured images from the property information management server 20.

[0180] Furthermore, the decision unit 48 decides whether to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B. Here, the image management server 40 received a login from the image request program used when logging into the property information management server 20 in step S28, so the decision unit 48 determines that the login was via the property information management server 20. Therefore, the decision unit 48 decides to update the first tacit knowledge model 4004A. The process by which the decision unit 48 determines which model to update is explained in Figure 17.

[0181] S34: The transmitting / receiving unit 11 of the terminal device 10 receives the inspection information and the request for captured images along with the model identification information. For example, the image management server 40 notifies the terminal device 10 of the URL of the property information management server 20 and redirects it. As a result, the transmitting / receiving unit 11 of the terminal device 10 sends the inspection information and the request for captured images of the item to the property information management server 20, specifying the model identification information.

[0182] S35: The transmitting / receiving unit 21 of the property information management server 20 receives requests for inspection information and captured images of items, and the storage / reading unit 29 searches the property information management DB 2001 using the model identification information to obtain the inspection information and captured images. The transmitting / receiving unit 21 transmits the inspection information and captured images to the terminal device 10.

[0183] S36: When the transmitting / receiving unit 11 of the terminal device 10 receives inspection information and captured images, it specifies the model identification information and sends the inspection information and captured images to the image management server 40.

[0184] S37: The transmitting / receiving unit 41 of the image management server 40 receives inspection information and captured images. The storage / reading unit 49 stores the inspection information and captured images in the image information management DB 4001, associating them with the model identification information. The judgment unit 43 obtains the caption comment identified by the model identification information from the caption model 4003 and determines the degree of association between the caption comment and the comments included in the input information received in step S32. The judgment unit 43 may determine the degree of association between the acquired caption comment and all the comments included in the input information received in step S32, or it may divide the comments included in the input information received in step S32 into multiple parts and determine the degree of association between the acquired caption comment and each divided comment.

[0185] S38: The update unit 46 updates the caption model 4003 by associating the comments deemed highly relevant in step S37 with the model identification information as caption comments. The update unit 46 also updates the first tacit knowledge model 4004A determined by the decision unit 48. The update unit 46 updates the first tacit knowledge model 4004A using the comments and inspection information deemed less relevant in step S37, along with the related three-dimensional image information of the item (identified in step S32) and captured images, as training data. In other words, the correspondence between the three-dimensional image information and captured images of the item and the comments and inspection information is learned. Features are extracted from the three-dimensional image information and captured images of the item using several feature extraction models suitable for images, such as CNNs. Features represent, for example, what is pictured where, or the content of the work being performed. Therefore, the first tacit knowledge model 4004A can learn the correspondence between the features of the three-dimensional image information and captured images of the item and the comments and inspection information.

[0186] Note that both comments and inspection information are not necessarily required; the first tacit knowledge model 4004A can be updated with at least one of them.

[0187] Figure 17 is a flowchart illustrating how the decision unit 48 determines whether to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B. In Figure 17, the decision unit 48 determines whether it has logged in to the image management server 40 via the property information management server 20 (S111). The decision unit 48 can determine whether it has logged in to the image management server 40 via the property information management server 20 by whether the login was performed using an image request program distributed from the property information management server 20.

[0188] If the decision in step S111 is Yes, the decision unit 48 decides to update the first tacit knowledge model 4004A (S112).

[0189] If the decision in step S111 is No, the decision unit 48 decides to update the second tacit knowledge model 4004B (S113).

[0190] Although Figure 17 illustrates the process of updating a model, it can also be applied to selecting a model used to generate text information.

[0191] <<Screen example>> The property selection screen 200 in the learning phase can be the same as in Figure 9, and the property management screen 210 can be the same as in Figure 10. On the other hand, the property display screen 220 in Figure 11 will be as shown in Figure 18.

[0192] Figure 18 shows a property display screen 330 that includes property management information and three-dimensional image information. The property display screen 330 includes a first display area 214 and a second display area 215. The first display area 214 displays an item list 216 and the area (floor area) 224. The items in this room are the ceiling, floor, walls, and door. The second display area 215 of the property display screen 330 displays three-dimensional image information 222 and a layout diagram 231 of these items, such as a table. The layout diagram 231 is a plan view showing the location of each item placed in the property. The user has entered input information 241 that can become an implicit knowledge comment, so the second display area 215 displays the input information 241, "This table has an unstable center of gravity, so it is best not to place anything heavier than 50 kg on it," after the user clicked on the three-dimensional image information 223 of the table. The image management server 40 can update the first implicit knowledge model 4004A using this input information 241 and inspection information. Please note that the property information, specifically the area (floor area) of 224, can be used as a caption comment.

[0193] When the user presses the information update button 226, the first tacit knowledge model 4004A is updated. When the information display button 225 is pressed, text information is generated based on the tacit knowledge comments generated by the first tacit knowledge model 4004A.

[0194] ● Inference Phase (Text Information Generation) Next, we will describe the text information generation process using the first tacit knowledge model 4004A.

[0195] Figure 19 is a sequence diagram showing an example of the text information generation process. The explanation of Figure 19 mainly focuses on the differences from Figure 16. The processes in steps S41 to S56 are the same as in Figure 16. However, in step S51, the user enters a question and presses the information display button 225 (see Figure 20).

[0196] S57: The decision unit 48 decides whether to use the first tacit knowledge model 4004A or the second tacit knowledge model 4004B to generate text information. Here, the image management server 40 received a login from the image request program used when logging into the property information management server 20 in step S48, so the decision unit 48 determines that the login was via the property information management server 20. Therefore, the decision unit 48 decides to generate text information using the first tacit knowledge model 4004A.

[0197] When the image management server 40's transmitting / receiving unit 41 receives inspection information, the storage / reading unit 49 identifies the item using model identification information and saves the latest inspection information and captured image in the image information management DB 4001. The processing unit 47 requests the text information generation unit 45 to generate text information. The text information generation unit 45 obtains tacit knowledge comments corresponding to the three-dimensional image information and captured image of the item from the first tacit knowledge model 4004A. The first tacit knowledge model 4004A extracts features of the three-dimensional image information and captured image of the item and can identify at least one of the inspection information or input information (comments) corresponding to these features as a tacit knowledge comment. The three-dimensional image information of the item is the three-dimensional image information of the item identified by the user in step S51. The user may specify the three-dimensional image information of multiple or all items.

[0198] S58: Next, the text information generation unit 45 obtains text information created by the large-scale language model 4005 using the tacit knowledge comments, input information (question text), and check information. The large-scale language model 4005 can use these to generate more detailed text information. The text information generation unit 45 may also convert the audio information contained in the input information into text information, and the text information generated by the text information generation unit 45 may be either audio information or text information.

[0199] The text information generation unit 45 may generate text information without using any inspection information or question text. Alternatively, the text information generation unit 45 may generate fixed questions internally within the system and use these fixed questions. In this case, the question text is not visible to the user. Or, the text information generation unit 45 may generate fixed questions internally within the system, display these fixed questions on the display unit for the user to select, and then use the selected question.

[0200] While inspection information is not mandatory, using it to generate text information from the large-scale language model 4005 provides more detailed information about the item. For example, if the inspection information includes the extent of damage to the item, it is possible to generate text information that includes appropriate responses based on the extent of the damage.

[0201] S59: The processing unit 47 associates the three-dimensional image information and captured image of the item corresponding to the model identification information with the text information and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to the second display area 215, which displays the text information, the three-dimensional image information of the item, and the captured image in association. The screen generation unit 42 may also perform an update process to add only the text information to the screen corresponding to the second display area 215. The transmitting / receiving unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10, and the transmitting / receiving unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0202] S60: The display control unit 13 of the terminal device 10 displays a text image screen 230 which includes a first display area 214 and a second display area 215 which includes text information (see Figure 21). Alternatively, the conversion unit 15 converts the received text information into audio information, and the audio control unit 14 plays the converted text information through the speaker 109a. If the received text information is audio information, the text information is played through the speaker 109a, or the conversion unit 15 converts the received text information into character information and displays the converted text information on the display 106a.

[0203] <<Example of the inference phase screen>> Of the screens displayed by the terminal device 10 during the inference phase, the property selection screen 200 may be the same as in Figure 9, and the property management screen 210 may be the same as in Figure 10. On the other hand, the property display screen 220 in Figure 11 will be as shown in Figure 20.

[0204] Figure 20 shows an example of the property display screen 340 during the inference phase. The property display screen 240 includes a first display area 214 and a second display area 215. Figure 20 has the same configuration as Figure 18, but the user presses the three-dimensional image information 223 of the table and inputs a question 234 as input information. The second display area 215 displays the input information (question 234) corresponding to the three-dimensional image information 223 of the table. For example, the question 234 in Figure 20 is "There is a scratch on the table, what should I do?" Along with such a question 234, the user presses the information display button 225 to request the generation of text information using the tacit knowledge model. As a result, the text information generation unit 45 generates text information using the first tacit knowledge model 4004A and the large-scale language model 4005.

[0205] Note that the screen shown in Figure 20, which displays property management information (a list of room numbers and item names) and three-dimensional image information of the property 222, is an example of the first display screen.

[0206] Figure 21 shows an example of a text image screen 230 containing text information. The text image screen 230 includes a first display area 214 and a second display area 215. In Figure 21, the user requested implicit knowledge comments for the table, so the table's three-dimensional image information 223 and captured images 232 and 233 are displayed in the second display area 215. The three-dimensional image information 223 was obtained from the image management server 40, and the captured images 232 and 233 were obtained from the property information management server 20.

[0207] Furthermore, the text image screen 230 contains text information 235. The text information 235 is displayed in the second display area 215, corresponding to the three-dimensional image information 223 of the table. The text information 235 is "If the scratch is less than 1 mm deep, it will be repaired with paint. If it is 1 mm or more, it will be polished." The text information 235 is generated by the large-scale language model 4005 from tacit knowledge comments, inspection information, and question sentences. For example, if scratches are detected in the photographed images 232 and 233 of an item, tacit knowledge comments regarding the scratches on this item are extracted. These tacit knowledge comments, question sentences regarding the scratches, and inspection information that identifies the current state of the scratches are input to the large-scale language model 4005, so that appropriate text information can be generated in response to questions regarding the current scratches.

[0208] <Effects of generating text information using inspection data> The effects of generating text information using inspection information, as in this embodiment, will be explained.

[0209] 1. Comparative Example 1 (Using a General Large-Scale Language Model) Question: The user asks, "How should I repair the cracks?" Tacit comment: Tape or filler can be used.

[0210] 2. Comparative Example 2 (Case of learning from three-dimensional image information) • Learning Phase Input image: Three-dimensional image information Comment: Use tape for wide cracks and filler for narrow cracks. • Inference Phase Input image: Three-dimensional image information Question: "How should I repair the cracks?" Implicit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter.

[0211] 3. This embodiment (three-dimensional image information and captured image) • Learning Phase Input image: Three-dimensional image information and captured image Inspection information: Cracks found in the tape at the corners. • Inference Phase Input image: Three-dimensional image information and captured image Question: "How should I repair the cracks?" Tacit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter. However, when applying tape to corners, there is a risk of cracking, so please apply with care. In other words, "When applying tape to corners, there is a risk of cracking, so please apply with care" is the effect of learning the inspection information.

[0212] 4. This embodiment (three-dimensional image information, captured image, input information) • Learning Phase Input image: Three-dimensional image information and captured image Input information: Use tape for wide cracks and filler for narrow cracks. Inspection information: Cracks found in the tape at the corners. • Inference Phase Input image: Three-dimensional image information and captured image Question: "How should I repair the cracks?" Tacit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter. However, please be careful when applying tape to corners as there is a risk of cracking. In other words, the effect of learning the inspection information is that "please be careful when applying tape to corners as there is a risk of cracking."

[0213] <<Direct login to the image management server>> Next, we will explain the case where a user logs in directly to the image management server 40. We will also explain the model update process in which the second tacit knowledge model 4004B learns from the data. ●Learning Phase (Model Update) Figure 22 is a sequence diagram showing an example of the model update process. The process in steps S121 to S126 can be the same as in Figure 13.

[0214] S127: The transmitting / receiving unit 11 of the terminal device 10 receives three-dimensional image information of the property display screen 350, and the display control unit 13 displays the property display screen 350 (see Figure 23). In this embodiment, since the terminal device 10 is not logged into the property information management server 20, the item list 216 placed in the property is not displayed. However, the screen generation unit 42 may use the image information management DB 4001 managed by the image management server 40 to display information equivalent to the item list 216. Next, the user identifies an arbitrary item from the three-dimensional image information of the property. Once the user identifies an item, they can request an update of the second tacit knowledge model 4004B. The input receiving unit 12 of the terminal device 10 accepts the operation to identify an item. The item may be identified, for example, by the coordinates clicked by the user, or model identification information may be identified by these coordinates.

[0215] The user inputs comments (text information, audio) about the item into the terminal device 10, as explained in Figures 6 and 7. These comments are sometimes referred to as input information. These comments may constitute tacit knowledge comments. Furthermore, the comments may include caption comments describing the item.

[0216] S128: When the user presses the information update button 226, the transmitting / receiving unit 11 of the terminal device 10 sends to the image management server 40 a message indicating that the information update button 226 has been pressed, along with information identifying the item and the input information. The transmitting / receiving unit 41 of the image management server 40 receives these messages. The decision unit 48 decides whether to update the first tacit knowledge model 4004A or the second tacit knowledge model 4004B. The decision unit 48 determines that the login in step S121 is not a login using the image request program distributed from the property information management server 20, and therefore decides to update the second tacit knowledge model 4004B.

[0217] S129: The determination unit 43 obtains a caption comment identified by the model identification information from the caption model 4003 and determines the degree of association between the caption comment and the comments included in the input information received in step S128. The determination unit 43 may determine the degree of association between the obtained caption comment and all the comments included in the input information received in step S128, or it may divide the comments included in the input information received in step S128 into multiple parts and determine the degree of association between the obtained caption comment and each divided comment.

[0218] S130: The update unit 46 updates the caption model 4003 by associating the comments deemed highly relevant in step S129 with the model identification information as caption comments. The update unit 46 also updates the second tacit knowledge model 4004B using the comments deemed less relevant in step S129 and the related three-dimensional image information of the item (identified in step S128) as training data. In other words, the correspondence between the three-dimensional image information of the item and the comments is learned. Features are extracted from the three-dimensional image information of the item using several feature extraction models suitable for images, such as CNNs. Features represent, for example, what is pictured where, or the content of the work being performed. Therefore, the second tacit knowledge model 4004B can learn the correspondence between the features of the three-dimensional image information of the item and the comments.

[0219] <<Screen example>> The property selection screen 200 displayed by the terminal device 10 during the learning phase may be the same as in Figure 9. Figure 10 is a screen generated by the property information management server 20 and is therefore not displayed. The property display screen 350 of this embodiment will be described based on Figure 23.

[0220] Figure 23 shows the item display screen 350 of this embodiment. Comparing Figure 23 with Figure 18, the item list 216 is not displayed. This is because the terminal device 10 logged directly into the image management server 40. The user presses the three-dimensional image information 223 of the table, enters input information 241 that can become tacit knowledge comments, and presses the information update button 226. This starts the update of the second tacit knowledge model 4004B in the image management server 40.

[0221] Note that the screen shown in Figure 23, which displays the property's three-dimensional image information 222 but does not show property management information (such as the property's room number or item name), is an example of a second display screen.

[0222] ● Inference Phase (Text Information Generation) Next, we will describe the text information generation process using the second tacit knowledge model 4004B.

[0223] Figure 24 is a sequence diagram showing an example of the text information generation process. The explanation of Figure 24 mainly focuses on the differences from Figure 22. The processes in steps S141 to S148 are the same as in Figure 22. However, in step S147, the user enters a question and presses the information display button 225 (see Figure 25).

[0224] S149: When the user presses the information display button 225, the transmitting / receiving unit 11 of the terminal device 10 transmits to the image management server 40 that the information display button 225 has been pressed, along with information identifying the item and input information. The transmitting / receiving unit 41 of the image management server 40 receives these. The decision unit 48 decides whether to use the first tacit knowledge model 4004A or the second tacit knowledge model 4004B to generate the text information. The decision unit 48 determines that the login in step S141 is not a login using the image request program distributed from the property information management server 20, and therefore decides to use the second tacit knowledge model 4004B.

[0225] The processing unit 47 requests the text information generation unit 45 to generate text information. The text information generation unit 45 obtains tacit knowledge comments corresponding to the three-dimensional image information of the item from the second tacit knowledge model 4004B. The second tacit knowledge model 4004B can extract features of the three-dimensional image information of the item and identify inspection information corresponding to these features as tacit knowledge comments.

[0226] S150: Next, the text information generation unit 45 obtains text information created by the large-scale language model 4005 using the tacit knowledge comments and input information (question text). The large-scale language model 4005 can use these to generate more detailed text information. The text information generation unit 45 may also convert the audio information contained in the input information into text information, and the text information generated by the text information generation unit 45 may be either audio information or text information.

[0227] The text information generation unit 45 may generate text information without using any question text. Alternatively, the text information generation unit 45 may generate fixed questions internally within the system and use those fixed questions. In this case, the question text is not visible to the user. Or, the text information generation unit 45 may generate fixed questions internally within the system, display the fixed questions on the display unit for the user to select, and use the selected question.

[0228] The processing unit 47 associates the three-dimensional image information of the item corresponding to the model identification information with the text information and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to a second display area 215 that displays the text information and the three-dimensional image information of the item in association. The screen generation unit 42 may also perform an update process to add only the text information to the screen corresponding to the second display area 215.

[0229] S151: The transmitting / receiving unit 41 of the image management server 40 transmits the generated text information along with the screen information of the text image screen 370 to the terminal device 10. The transmitting / receiving unit 11 of the terminal device 10 receives the screen information of the text image screen 370 and the text information transmitted from the image management server 40.

[0230] S152: The display control unit 13 of the terminal device 10 displays a text image screen 370 containing text information (see Figure 26). Alternatively, the conversion unit 15 converts the received text information into audio information, and the audio control unit 14 plays the converted text information through the speaker 109a. If the received text information is audio information, the text information is played through the speaker 109a, or the conversion unit 15 converts the received text information into character information and displays the converted text information on the display 106a.

[0231] <<Example of the inference phase screen>> Of the screens displayed by the terminal device 10 during the inference phase, the property selection screen 200 can be the same as in Figure 9, and the property management screen 210 in Figure 10 is not displayed. On the other hand, the property display screen 220 in Figure 11 becomes as shown in Figure 25.

[0232] Figure 25 shows an example of the property display screen 360 during the inference phase. Figure 25 has the same configuration as Figure 20, but the item list 216 is not displayed. This is because the terminal device 10 has not logged into the property information management server 20. The user presses the three-dimensional image information 223 of the table to input a question sentence 234 and presses the information display button 225 to request the generation of text information using the tacit knowledge model. As a result, the text information generation unit 45 generates text information using the second tacit knowledge model 4004B and the large-scale language model 4005.

[0233] Note that the screen shown in Figure 25, which displays the property's three-dimensional image information 222 but does not show property management information (such as the property's room number or item name), is an example of a second display screen.

[0234] Figure 26 shows an example of a text image screen 370 containing text information. The text image screen 370 has text information 236. This text information 236 is "Scratches will be addressed with paint or polishing." Text information 236 was generated by the large-scale language model 4005 from the tacit knowledge comment generated by the second tacit knowledge model 4004B and the input information (question text). Therefore, even if scratches are detected in the three-dimensional image information of the item, the tacit knowledge comment does not reflect inspection information. Furthermore, the large-scale language model 4005 does not use inspection information to generate text information.

[0235] Therefore, comparing the text information 236 in Figure 26 with the text information 235 in Figure 21, text information 236 is general text information about table scratches and is less detailed than text information 235. However, text information 236 is highly versatile information regarding table scratches.

[0236] <Example of an image management server retrieving inspection information from a property information management server> In Figures 16 and 19, the terminal device 10 acquires inspection information from the property information management server 20, and the image management server 40 acquires it from the terminal device 10. However, it is also possible for the image management server 40 to directly acquire inspection information from the property information management server 20.

[0237] Figure 27 is a sequence diagram showing an example of the process in which the image management server 40 communicates with the property information management server 20 to update the model. While Figure 27 primarily explains the differences from Figure 16, the sequence diagram in Figure 19 can be similarly modified. First, the processes in steps S21 to S32 are the same as in Figure 16.

[0238] S33: The transmitting / receiving unit 41 of the image management server 40 receives information that the information update button 226 has been pressed, as well as information identifying the item and input information. The storage / reading unit 49 identifies the model identification information of the item with the closest location information from the image information management DB 4001 based on the information identifying the item (in this case, location information). The processing unit 47 requests the transmitting / receiving unit 41 to acquire inspection information and captured images. The transmitting / receiving unit 41 calls the API of the property information management server 20, specifying the model identification information and sending the request for inspection information and captured images to the property information management server 20. Note that if the information identifying the item is model identification information, searching the image information management DB 4001 is unnecessary.

[0239] S36: The transmitting / receiving unit 21 of the property information management server 20 receives requests for inspection information and captured images. The storage / reading unit 29 searches the property information management DB 2001 using the model identification information and obtains the inspection information and captured images. The transmitting / receiving unit 21 sends the inspection information and captured images to the image management server 40.

[0240] The subsequent processing can be the same as in Figure 16. Also, in Figure 27, the process by which the image management server 40 acquires inspection information and captured images from the property information management server 20 was explained using a sequence diagram during model update as an example, but the same process applies when generating text information in Figure 19.

[0241] <Criteria for users to decide which login to use> From the user's perspective, the text information they receive differs depending on whether they log in to the image management server 40 or the property information management server 20. Therefore, we will explain the criteria that users use to decide which server to log in to.

[0242] As described above, the first tacit knowledge model 4004A and the second tacit knowledge model 4004B have the following differences. • The first tacit knowledge model 4004A can generate expert knowledge and text information specific to particular items. • The second tacit knowledge model 4004B can generate expert knowledge and general text information applicable to similar (same-category) items in general.

[0243] For example, if a specific item is a turbo chiller used by a customer, then because it is an item owned by the customer, there is an accumulation of inspection history and know-how regarding that item. The user decides to use this information and the first tacit knowledge model 4004A to generate highly accurate text information (answers) specifically for that particular item.

[0244] On the other hand, while such highly accurate information does not exist for all similar items (refrigeration machines in the category of turbo chillers), specialized knowledge about turbo chillers in general does exist. Therefore, in this case, it is thought that the user would use the second tacit knowledge model 4004B to generate general text information (answers) applicable to all similar items. In this way, the user can decide whether or not to use inspection information to generate text information depending on how detailed the information required.

[0245] Furthermore, instead of the user making the decision, the decision unit 48 may decide automatically (without confirming with the user) or semi-automatically (by recommending to the user and asking for confirmation) whether to use inspection information and captured images to generate text information. For example, if the question sentence included in the input information (voice, text) concerns inspection information of an item, the decision unit 48 may decide to use the first tacit knowledge model 4004A because the second tacit knowledge model 4004B may not be able to generate appropriate text information.

[0246] Furthermore, there may be three or more types of tacit knowledge models, not just two, and users may be able to select the tacit knowledge model they prefer.

[0247] <Multimodal> Describe some examples of combinations of input information and tacit knowledge comments. In the above-described model, it is assumed that it is a large language model. However, in the present embodiment, a multimodal model that inputs a plurality of data formats (images, texts, gestures, etc.) and outputs in a predetermined data format can be used. · When the input information is a character string and content other than text information is generated as tacit knowledge comments Input a character string to generate an image Input a character string to generate a video Input a character string to generate audio Input a character string to generate a 3D model · When the input information includes a character string and something other than a character string, and text information is generated as tacit knowledge comments Input an image and a character string to generate text information Input a 3D model and a character string to generate text information Input audio and a character string to generate text information · When the input information includes a character string and something other than a character string, and content other than text information is generated as tacit knowledge comments Input an image and a character string to generate an image Input a video and a character string to generate a video Input a 3D model and a character string to generate a 3D model Input audio and a character string to generate audio <Main effect> According to the present embodiment, the image management server 40 can selectively use the first tacit knowledge model 4004A learned with inspection information and the second tacit knowledge model 4004B learned without using inspection information. That is, for a user who logs in via the property information management server 20, the first tacit knowledge model 4004A can be used to generate detailed text information, and for a user who logs in directly to the image management server 40, the second tacit knowledge model 4004B can be used to generate highly versatile text information.

[0248] [Third Embodiment] In this embodiment, an information processing system 100 in which the image management server 40 performs a simulation will be described. Since the simulation is only possible on the image management server 40, this function is not provided when the terminal device 10 is connected to the property information management server 20.

[0249] In this embodiment, the hardware configuration diagram shown in Figure 2 and the functional block diagram shown in Figure 3, as described in the above embodiment, can be used as references.

[0250] <Action or process> Figure 28 is a sequence diagram illustrating the process by which the image management server 40 executes the simulation. The processes in steps S201 to S210 can be the same as in Figure 8.

[0251] S211: The transmitting / receiving unit 11 of the terminal device 10 receives screen information from the property display screen 380, and the display control unit 13 displays the property display screen 380 (see Figure 29). The user presses the simulation execution button 243 to perform a simulation related to the property or item. The input reception unit 12 receives the press.

[0252] S212: Since the simulation is a function of the image management server 40, the transmitting / receiving unit 11 logs out of the property information management server 20. This is because the image management server 40 will execute the simulation while the user is logged in only to the image management server 40.

[0253] S213: Next, the user enters a login operation into the terminal device 10. The image request program should prompt the user to log in to the image management server 40. This login is to the image management server 40. The input reception unit 12 of the terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful. Since the user has already logged in to the image management server 40 in step S208, the login in step S213 may be omitted.

[0254] S214: Upon successful login, the transmitting / receiving unit 11 of the terminal device 10 sends a simulation execution request to the image management server 40. Since the user will later log in to the property information management server 20, the transmitting / receiving unit 11 may send the URL of the property information management server 20 to the image management server 40 for redirection.

[0255] S215: The transmitting / receiving unit 41 of the image management server 40 receives a request to execute a simulation. The simulation unit 52 executes the simulation. There are no particular restrictions on the content of the simulation, but the image management server 40 may execute any simulation it is capable of executing, or the user may specify one.

[0256] S216: The screen generation unit 42 of the image management server 40 generates the simulation result screen 390, and the transmission / reception unit 41 of the image management server 40 transmits the screen information of the simulation result screen 390 to the terminal device 10.

[0257] S217: The transmitting / receiving unit 11 of the terminal device 10 receives screen information from the simulation result screen 390, and the display control unit 13 displays the simulation result screen 390 (see Figure 30).

[0258] S218: After the simulation is complete, the user inputs a screen return command to the terminal device 10 to display the property display screen 380. The input reception unit 12 receives the input.

[0259] S219: In order for the terminal device 10 to connect to the property information management server 20, the transmitting / receiving unit 11 logs out from the image management server 40.

[0260] S220: The user enters a login operation into the terminal device 10. This login is for the property information management server 20. The input reception unit 12 of the terminal device 10 accepts the login operation. Any existing login method may be used. Here, we assume that the login was successful. Since the user will later log in to the image management server 40, the sending / receiving unit 11 may send the URL of the image management server 40 to the image management server 40 for redirection.

[0261] S221: The transmitting / receiving unit 11 of the terminal device 10 sends a request for property management information to the property information management server 20, specifying the property identification information.

[0262] S222: The transmitting / receiving unit 21 of the property information management server 20 receives a request for property management information, and the storage / reading unit 29 searches the property information management DB 2001 using the property identification information. The screen generation unit 22 of the property information management server 20 generates a property management screen 210 (see Figure 10) that displays the property management information, and the transmitting / receiving unit 21 transmits the screen information of the property management screen 210 to the terminal device 10.

[0263] Furthermore, the transmitting / receiving unit 21 transmits an image request program to the terminal device 10 in response to a request for property management information, so that the terminal device 10 can acquire three-dimensional image information.

[0264] S222: The transmitting / receiving unit 11 of the terminal device 10 receives screen information and an image request program for the property management screen 210. The display control unit 13 displays the property management screen 210. As a result, the property management information is displayed.

[0265] S223: The user enters a login operation into the terminal device 10. The image request program should prompt the user to log in to the image management server 40. This login is for the image management server 40. The input reception unit 12 of the terminal device 10 accepts the login operation. The login operation may be omitted by using single sign-on or similar methods.

[0266] S224: Since the three-dimensional image information of the property has been displayed previously, even if the user does not perform an operation to request the three-dimensional image information of the property, the transceiver unit 11 of the terminal device 10 can transmit a request for the three-dimensional image information of the property by specifying the property identification information of the property to the image management server 40, but the user may specify the property identification information again. Also, the transceiver unit 11 may transmit the property management information acquired from the property information management server 20. This is for the image management server 40 to display the property management information and the three-dimensional image information of the property on the terminal device in one screen.

[0267] S225: The transceiver unit 41 of the image management server 40 receives a request for the three-dimensional image information of the property and the property management information. The storage / reading unit 49 of the image management server 40 searches the image information management DB 4001 with the property identification information and acquires the three-dimensional image information of each item. The processing unit 47 requests the screen generation unit 42 to generate a screen including the property management information and the three-dimensional image information of the property. The screen generation unit 42 generates a property display screen 380 on which the property management information and the three-dimensional image information of each item are arranged. The transceiver unit 41 transmits the screen information of the property display screen 380 to the terminal device 10. The three-dimensional image information of each item is the three-dimensional image information in which all the items included in the property are arranged in the property, and the user can arbitrarily change the viewpoint.

[0268] S226: The transceiver unit 11 of the terminal device 10 receives the screen information of the property display screen 380, and the display control unit 13 displays the property display screen 380 (see FIG. 二十九).

[0269] <<Screen Example>> FIG. 二十九 is an example of the property display screen 380 of the present embodiment. In the description of FIG. 二十九, mainly the differences from FIG. 18 will be described. The property display screen 380 of FIG. 二十九 has a simulation execution button 243. Also, the input information (question text 234) is not input. When the user presses the three-dimensional image information 223 of the table and then presses the simulation execution button 243, the image management server 40 executes the simulation.

[0270] It should be noted that the "図29" in the original text is likely a reference to a specific figure number in a Japanese context. Here, it is directly translated as "FIG. 二十九" as per the rules. If there is a more specific English equivalent for this figure number in the overall context, it should be adjusted accordingly.Note that the screen shown in Figure 29, which displays property management information (such as the property's room number and item name) and the property's three-dimensional image information 222, is an example of the first display screen.

[0271] Figure 30 shows an example of the simulation results screen 390. The simulation results screen 390 displays the simulation results performed on the three-dimensional image information of the property. Property management information is not displayed on the simulation results screen 390. This is because the terminal device 10 logged directly into the image management server 40.

[0272] Also, the diagram shows three simulation results 251-253, but these numbers are just examples. For example, simulation result 251 specifies a cylinder and points out, "The cylinder is in the path of movement." Simulation result 252 specifies a cylinder and points out, "It may not pass through the opening." Simulation result 253 specifies a storage box and points out, "It may be exposed to the air conditioner's airflow."

[0273] Furthermore, the simulation results screen 390 has a button 244 that returns to the property display screen 380. Pressing the button 244 that returns to the property display screen 380 corresponds to a screen return instruction. By pressing the button 244 that returns to the property display screen 380, the user can transition to the property display screen 380 shown in Figure 29.

[0274] Note that the screen shown in Figure 30, where property management information (such as the room number and item name of the property) is not displayed but the three-dimensional image information 222 of the property is displayed, is an example of a second display screen.

[0275] [Fourth Embodiment] In this embodiment, an image management server 40 that generates images from captured images and text information will be described.

[0276] Figure 31 is an example functional configuration diagram illustrating the functions of the image management server 40, the property information management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment. Note that the explanation of Figure 31 mainly focuses on the differences from Figure 3.

[0277] The image management server 40 in Figure 31 further includes an image generation unit 53, and an image generation model 4006 is constructed in the storage unit 4000 of the image management server 40. The other configurations may be the same as in Figure 3.

[0278] The image generation unit 53 is an example of an image generation means, and is implemented by instructions from the CPU 401 shown in Figure 2. It inputs text data or text data and images to the image generation model 4006 to generate image information.

[0279] Image generation model 4006 is a machine learning model (generative AI) that generates images from text data, or from text data and images. Image generation model 4006 is trained using training data that includes text data and images. This training data includes, for example, training text data, or text data and images as input, and ground truth images as output. For example, the image generation model 4006 may be trained so that the images it generates when inputting text data, or text data and images, approach the ground truth images included in the training data.

[0280] <Learning Phase> The processing in the learning phase may be the same as in Figure 16. In step S38, the update unit 46 updates the first tacit knowledge model 4004A to learn a correspondence between inputting comments and inspection information that were determined to have a low degree of relevance in step S37, and outputting three-dimensional image information or captured images of the item. Alternatively, the update unit 46 updates the first tacit knowledge model 4004A to learn a correspondence between inputting comments, inspection information, and three-dimensional image information (or captured images) of the item, and outputting captured images (or three-dimensional image information).

[0281] <Inference Phase (Generation of Text Information)> Figure 32 is an example sequence diagram showing the process of generating text and image information. Note that the explanation of Figure 32 mainly focuses on the differences from Figure 19. Step S58-2 is added in Figure 32.

[0282] S58-2: The image generation unit 53 inputs the captured image and the text information created by the large-scale language model to the image generation model 4006 to generate image information. The image generation unit 53 may also obtain the image information created by the image generation model 4006 using the text information created by the large-scale language model without using the captured image.

[0283] The memory / reading unit 49 associates the captured images and inspection information stored in the image information management DB 4001 in step S56 with the text information created by the large-scale language model and the image information created by the image generation model 4006 and stores (or overwrites) them in the image information management DB 4001.

[0284] S59: The processing unit 47 associates the three-dimensional image information of the item corresponding to the model identification information with the generated image information and text information, and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the three-dimensional image information of the item, the generated image information and text information. The transmitting / receiving unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0285] <Example of the inference phase screen> Figure 33 shows the generated image information displayed on the text image screen 280. The explanation of Figure 33 mainly focuses on the differences from Figure 21.

[0286] The text image screen 280 in Figure 33 displays generated images 281 and 282. Generated images 281 and 282 are not the captured images 232 and 233 described in Figure 21, but rather generated images produced by the image generation model 4006 based on the captured images 232 and 233 and the text information 235. For this reason, generated images 281 and 282 have markers 283 and 284 that indicate the location of the wound.

[0287] <Main effects> According to this embodiment, when a user executes a function that only the image management server 40 has, such as a simulation function, the terminal device 10 logs out of the property information management server 20 and logs in to the image management server 40. Therefore, the functions that the image management server 40 has can be provided to the user while they are logged into the image management server 40.

[0288] [Other application examples] The present invention is not limited to the embodiments specifically disclosed above, and various modifications and changes are possible without departing from the scope of the claims. It goes without saying that the image management server 40 described in this embodiment is just one example, and there are various system configurations depending on the application and purpose.

[0289] For example, in this embodiment, we described an example in which an tacit knowledge model for industries such as civil engineering and construction answers a question, but the tacit knowledge model can be used in any industry where tacit knowledge is effective, such as healthcare, dental care, and investment decisions.

[0290] Furthermore, in this embodiment, the large-scale language model 4005 generates text information based on tacit knowledge comments, but it is also possible to use tacit knowledge comments as text information without using the large-scale language model 4005.

[0291] Furthermore, the first tacit knowledge model 4004A may also be one that takes three-dimensional image information, captured images, and inspection information as inputs, and learns tacit knowledge comments by outputting the input information. In other words, it may take different forms of information, such as images and text, as inputs.

[0292] Furthermore, the image management server 40 may generate two sets of text information using not only one of the first tacit knowledge model 4004A or the second tacit knowledge model 4004B, but both. In other words, the image management server 40 only needs to generate text information using at least one of the first tacit knowledge model 4004A or the second tacit knowledge model 4004B.

[0293] Furthermore, although this embodiment describes a client-server type information processing system 100, the functions of the image management server 40 may be installed as an application on the terminal device 10. In other words, users may use the functions of this embodiment in a standalone format.

[0294] Furthermore, the configuration examples shown in Figure 3 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the image management server 40. The present invention is not limited by the way the processing units are divided or their names. The processing of the image management server 40 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processing.

[0295] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0296] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the image management server 40 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0297] Furthermore, the image management server 40 can combine the disclosed processing steps in various ways. Each element of the image management server 40 may be combined into a single device or divided into multiple devices. Also, each processing performed by the image management server 40 may be performed by the terminal device 10.

[0298] <Nature> [Aspect 1] An information processing system comprising: a first server for managing two-dimensional images of an article and information relating to the article; a second server for managing three-dimensional image information of the article; and a terminal device capable of communicating with the first server and the second server, The aforementioned terminal device is A first display screen is displayed which includes information about the article received from the first server when a session is established with the first server, and three-dimensional image information of the article received from the second server when a session is established with both the first and second servers. The system includes a display control unit that displays a second display screen containing three-dimensional image information of the article received from the second server when a session is established with the second server, The second server described above is Processing based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information of the article for the first display screen, or A processing unit that performs processing on the second display screen based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information. An information processing system having [a certain feature]. [Aspect 2] The display control unit, Based on the input information received by the input receiving unit and the three-dimensional image information of the article, the processing result executed by the processing unit is displayed on the first display screen. Based on the input information received by the input receiving unit and the three-dimensional image information of the article, the processing result executed by the processing unit is displayed on the second display screen. The information processing system according to claim 1. [Aspect 3] The processing unit measures the distance between two points received by the input receiving unit with respect to the three-dimensional image information of the article displayed on the first display screen. The processing unit measures the distance between two points specified by the input information with respect to the three-dimensional image information of the article displayed on the second display screen. The information processing system according to claim 2. [Aspect 4] The aforementioned processing unit, The first server performs a process to associate the two-dimensional image of the article transmitted by the first server with the three-dimensional image information of the article and the generated information generated based on the two-dimensional image and the three-dimensional image information. The display control unit, The two-dimensional image of the article, the three-dimensional image information of the article, and the generated information obtained from the second server are displayed on the first display screen. The information processing system described in Embodiment 2. [Aspect 5] The second server described above is The first server obtains the information about the article and the two-dimensional image transmitted by the first server, A first model that has learned the correspondence between the three-dimensional image information and the two-dimensional image of the article and information relating to the article, or the correspondence between the three-dimensional image information and the two-dimensional image of the article and information relating to the article and input information input to the terminal device, A second model that has learned the correspondence between the three-dimensional image information of the aforementioned article and the input information input to the terminal device, The terminal device generates three-dimensional image information of the selected article, the two-dimensional image, information about the article, and text information about the article using the first model, or A determination unit that determines whether to generate text information about the article using the three-dimensional image information of the article selected by the terminal device and the second model, An information processing system according to any one of embodiments 1 to 4. [Aspect 6] The determination unit determines whether to use the first model or the second model depending on whether the terminal device logged in directly to the second server or logged in to the first server and then logged in to the second server. The information processing system described in aspect 5. [Aspect 7] The aforementioned processing unit, The input information received by the input receiving unit for the first display screen, The information relating to the article and the two-dimensional image obtained from the first server, or the information relating to the article and the two-dimensional image obtained by the terminal device from the first server and transmitted to the second server, The three-dimensional image information of the aforementioned article, The first process based on is executed. The information processing system described in aspect 5. [Aspect 8] The first process described above is: The process includes generating text information based on the three-dimensional image information of the article, the two-dimensional image, information relating to the article, input information received by the input receiving unit for the first display screen, and the first model. The information processing system described in Embodiment 7. [Aspect 9] The first process described above is: This process includes updating the first model by learning the correspondence between the three-dimensional image information and the two-dimensional image of the article, information relating to the article, and input information received by the input receiving unit for the first display screen. The information processing system described in aspect 8. [Aspect 10] The aforementioned processing unit, The input information received by the input receiving unit for the second display screen, The three-dimensional image information of the aforementioned article, Perform a second process based on the above. The information processing system described in aspect 5. [Aspect 11] The second process described above is: The process includes generating text information based on the three-dimensional image information of the article, the input information received by the input receiving unit for the first display screen, and the second model. The information processing system described in aspect 10. [Aspect 12] The second process described above is: This process includes learning the correspondence between the three-dimensional image information of the article and the input information received by the input receiving unit for the first display screen, and updating the second model. The information processing system described in aspect 10. [Aspect 13] The aforementioned processing unit, The information processing system according to embodiment 1, which performs a third process based on three-dimensional image information of the article managed by the second server. [Aspect 14] The third process described above is: This is a process that performs a simulation based on the three-dimensional image information of the aforementioned article. The information processing system described in Embodiment 13. [Aspect 15] With the terminal device logged into the first server, When the input receiving unit accepts the execution of the simulation, the terminal device logs out of the first server. Log in to the second server and request the execution of the simulation. The information processing system described in Embodiment 14. [Aspect 16] The three-dimensional image information of the article is a two-dimensional projection of the three-dimensional model shape of the article, and the three-dimensional image information of the article can be displayed by changing the viewpoint. An information processing system as described in any one of the descriptions 1 to 15. [Aspect 17] The information regarding the aforementioned item is information entered by the user after inspecting the condition of the item. An information processing system as described in any one of the descriptions 1 to 16. [Explanation of Symbols]

[0299] 10 Terminal devices 20 Property Information Management Server 40 Image Management Server 100 Information Processing Systems [Prior art documents] [Patent Documents]

[0300] [Patent Document 1] Patent No. 6954410

Claims

1. An information processing system comprising: a first server for managing two-dimensional images of an article and information relating to the article; a second server for managing three-dimensional image information of the article; and a terminal device capable of communicating with the first server and the second server, The aforementioned terminal device is A first display screen is displayed which includes information about the article received from the first server when a session is established with the first server, and three-dimensional image information of the article received from the second server when a session is established with both the first and second servers. The system includes a display control unit that displays a second display screen containing three-dimensional image information of the article received from the second server when a session is established with the second server, The second server is, Processing based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information of the article for the first display screen, or A processing unit that performs processing on the second display screen based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information. An information processing system having

2. The display control unit, Based on the input information received by the input receiving unit and the three-dimensional image information of the article, the processing result executed by the processing unit is displayed on the first display screen. Based on the input information received by the input receiving unit and the three-dimensional image information of the article, the processing result executed by the processing unit is displayed on the second display screen. The information processing system according to claim 1.

3. The processing unit measures the distance between two points received by the input receiving unit with respect to the three-dimensional image information of the article displayed on the first display screen. The processing unit measures the distance between two points specified by the input information with respect to the three-dimensional image information of the article displayed on the second display screen. The information processing system according to claim 2.

4. The aforementioned processing unit, The first server performs a process to associate the two-dimensional image of the article transmitted by the first server with the three-dimensional image information of the article and the generated information generated based on the two-dimensional image and the three-dimensional image information. The display control unit, The two-dimensional image of the article, the three-dimensional image information of the article, and the generated information obtained from the second server are displayed on the first display screen. The information processing system according to claim 2.

5. The second server is, The first server obtains the information about the article and the two-dimensional image transmitted by the first server, A first model that has learned the correspondence between the three-dimensional image information and the two-dimensional image of the article and information relating to the article, or the correspondence between the three-dimensional image information and the two-dimensional image of the article and information relating to the article and input information input to the terminal device, A second model that has learned the correspondence between the three-dimensional image information of the aforementioned article and the input information input to the terminal device, The terminal device generates three-dimensional image information of the selected article, the two-dimensional image, information about the article, and text information about the article using the first model, or A determination unit that determines whether to generate text information relating to the article using the three-dimensional image information of the article selected by the terminal device and the second model, An information processing system according to any one of claims 1 to 4.

6. The determination unit determines whether to use the first model or the second model depending on whether the terminal device logged in directly to the second server or logged in to the first server and then logged in to the second server. The information processing system according to claim 5.

7. The aforementioned processing unit, The input information received by the input receiving unit for the first display screen, Information relating to the article and the two-dimensional image obtained from the first server, or information relating to the article and the two-dimensional image obtained by the terminal device from the first server and transmitted to the second server, The three-dimensional image information of the aforementioned article, The first process based on is executed. The information processing system according to claim 5.

8. The first process is, The process includes generating text information based on the three-dimensional image information of the article, the two-dimensional image, information relating to the article, input information received by the input receiving unit for the first display screen, and the first model. The information processing system according to claim 7.

9. The first process is, This process includes updating the first model by learning the correspondence between the three-dimensional image information and the two-dimensional image of the article, the information relating to the article, and the input information received by the input receiving unit for the first display screen. The information processing system according to claim 8.

10. The aforementioned processing unit, The input information received by the input receiving unit for the second display screen, The three-dimensional image information of the aforementioned article, The second process based on is executed. The information processing system according to claim 5.

11. The second process described above is: The process includes generating text information based on the three-dimensional image information of the article, the input information received by the input receiving unit for the first display screen, and the second model. The information processing system according to claim 10.

12. The second process described above is: This process includes learning the correspondence between the three-dimensional image information of the article and the input information received by the input receiving unit for the first display screen, and updating the second model. The information processing system according to claim 10.

13. The aforementioned processing unit, The information processing system according to claim 1, wherein a third process is performed based on the three-dimensional image information of the article managed by the second server.

14. The third process described above is: This is a process that performs a simulation based on the three-dimensional image information of the aforementioned article. The information processing system according to claim 13.

15. With the terminal device logged into the first server, When the input receiving unit accepts the execution of the simulation, the terminal device logs out of the first server. Log in to the second server and request the execution of the simulation. The information processing system according to claim 14.

16. The three-dimensional image information of the article is a two-dimensional projection of the three-dimensional model shape of the article, and the three-dimensional image information of the article can be displayed by changing the viewpoint. The information processing system according to claim 1.

17. The information regarding the aforementioned item is information entered by the user after inspecting the condition of the item. The information processing system according to claim 1.

18. The system displays a first display screen that includes a two-dimensional image of an article and information about the article received from the first server when a session is established with the first server, and a three-dimensional image of the article received from the second server when a session is established with both the first and second servers. A terminal device equipped with a display control unit that displays a second display screen including three-dimensional image information of the article received from the second server when a session is established with the second server, and A first server manages a two-dimensional image of the article and information relating to the article, and a second server that can communicate with the first server, Processing based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information of the article for the first display screen, or A processing unit that performs processing on the second display screen based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information. A second server having [a certain feature].

19. An information processing method performed by the second server in an information processing system including a first server that manages two-dimensional images of an article and information related to the article, a second server that manages three-dimensional image information of the article, and a terminal device capable of communicating with the first server and the second server, The aforementioned terminal device is A first display screen is displayed which includes information about the article received from the first server when a session is established with the first server, and three-dimensional image information of the article received from the second server when a session is established with both the first and second servers. When a session is established with the second server, a second display screen is displayed that includes three-dimensional image information of the item received from the second server. The second server is, Processing based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information of the article for the first display screen, or The process involves performing a step of processing the input information received by the input receiving unit of the terminal device and the three-dimensional image information on the second display screen. Information processing methods.

20. A program to be executed by the second server in an information processing system including a first server that manages information about an article, including a two-dimensional image of the article; a second server that manages three-dimensional image information of the article; and a terminal device capable of communicating with the first server and the second server, The aforementioned terminal device is The system displays a first display screen that includes information about an item received from the first server when a session is established with the first server, and three-dimensional image information of the item received from the second server when a session is established with both the first and second servers. When a session is established with the second server, a second display screen is displayed that includes three-dimensional image information of the item received from the second server. On the second server described above, Processing based on the input information received by the input receiving unit of the terminal device and the three-dimensional image information of the article for the first display screen, or A program that causes the terminal device to perform processing based on the input information received by the input receiving unit and the three-dimensional image information for the second display screen.

Images