Information processing system, first server, second server, information processing method, program

Abstract

To perform processing based on information managed by the first server and information managed by the second server, without adding processing functions to the first server. [Solution] The information processing system 100 includes a first server 20 that manages information about articles, including a two-dimensional image of the article, a second server 40 that manages three-dimensional image information of the article, and a terminal device 10. The terminal device displays a display screen that includes information about the article received from the first server and three-dimensional image information of the article received from the second server. The second server performs a process that associates the three-dimensional image information of the article with information about the article or generated information generated based on the information about the article, based on the information about the article transmitted by the first server and the three-dimensional image information of the article. The terminal device displays the three-dimensional image information of the article and the information about the article or generated information corresponding to the three-dimensional image information on the display screen.

Description

Technical Field

[0001] The present invention relates to an information processing system, a first server, a second server, an information processing method, and a program.

Background Art

[0002] There may be cases where a first server and a second server each manage related information. In such cases, a technique is known in which a terminal device displays information managed by the first server and information managed by the second server (see, for example, Patent Document 1).

[0003] Patent Document 1 discloses a system that displays related information of a property transmitted from a link information management system and a full-spherical image of the property transmitted from an image management system on a communication terminal.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention provides a technique for performing processing based on information managed by a first server and information managed by a second server without adding a processing function to the first server.

Means for Solving the Problems

[0005] In view of the above problems, the present invention provides an information processing system comprising: a first server for managing information about an article, including a two-dimensional image of the article; a second server for managing three-dimensional image information of the article; and a terminal device capable of communicating with the first server and the second server, wherein the terminal device includes a display control unit that displays a display screen including information about the article received from the first server and three-dimensional image information of the article received from the second server; the second server includes a processing unit that performs a process to associate the three-dimensional image information of the article with information about the article or generated information generated based on the information about the article, based on the information about the article transmitted by the first server and the three-dimensional image information of the article; and the display control unit of the terminal device displays the three-dimensional image information of the article and the information about the article or generated information corresponding to the three-dimensional image information, received from the second server, on the display screen. [Effects of the Invention]

[0006] Without adding processing functions to the first server, processing can be performed based on information managed by the first server and information managed by the second server. [Brief explanation of the drawing]

[0007] [Figure 1] This is an overall configuration diagram of an example of an information processing system. [Figure 2] This is a hardware configuration diagram of an example of an image management server, property information management server, or terminal device. [Figure 3] This is an example functional configuration diagram illustrating the functions of an image management server, a property information management server, and terminal devices within an information processing system. [Figure 4] This is a conceptual diagram showing an example of an image information management table according to this embodiment. [Figure 5] This is a conceptual diagram showing an example of a property management information management table according to this embodiment. [Figure 6] This is an explanatory diagram displayed by the terminal device during text information generation processing and model update processing. [Figure 7] This is an explanatory diagram displayed by the terminal device during text information generation processing and model update processing. [Figure 8] This sequence diagram shows an example of a process that generates screen information by arranging inspection information, three-dimensional image information, and captured images. [Figure 9] This diagram shows an example of a property management screen. [Figure 10] This diagram shows an example of a property management screen. [Figure 11] This is a diagram showing an example of a property display screen. [Figure 12] This is a diagram showing an example of an inspection image screen. [Figure 13] This diagram illustrates an example of switching between items and inspection information displayed in the second display area. [Figure 14] This sequence diagram shows an example of a process that generates screen information based on inspection information, three-dimensional image information, and captured images. [Figure 15] This is a sequence diagram showing an example of a model update process. [Figure 16] This figure shows an example of a property display screen shown by a terminal device. [Figure 17] This is an example sequence diagram illustrating the text information generation process. [Figure 18] This figure shows an example of a property display screen during the inference phase. [Figure 19] This figure shows an example of text information displayed on an inspection image screen. [Figure 20] This is an example functional configuration diagram illustrating the functions of an image management server, a property information management server, and terminal devices within an information processing system. [Figure 21] This is an example sequence diagram showing the process of generating text and image information. [Figure 22] This figure shows the generated image information displayed on the inspection image screen. [Modes for carrying out the invention]

[0008] Hereinafter, as an example of an embodiment for implementing the present invention, an information processing system and an information processing method performed by the information processing system will be described while referring to the drawings.

[0009] <Supplementary Note on Tacit Knowledge> In industries such as civil engineering and architecture, BIM / CIM implementation is underway for the purpose of addressing issues such as the declining birthrate and aging population, and improving labor productivity.

[0010] BIM is an abbreviation for Building Information Modeling. It is a solution for leveraging information in all processes from architectural design, construction to maintenance management, using a database of buildings that adds attribute data such as costs, finishes, and management information to a three-dimensional digital model of a building (hereinafter referred to as a 3D model) created on a computer.

[0011] CIM is an abbreviation for Construction Information Modeling. It is a solution for the civil engineering field (covering all infrastructure such as roads, electricity, gas, water supply, etc.) proposed following the example of BIM which has been promoted in the architectural field. Similar to BIM, CIM aims to improve and enhance the efficiency of a series of construction production systems by sharing information among relevant parties centered around the 3D model.

[0012] [[ID=第十九]]

[0013] Specifically, the 3D model restored by BIM and CIM can be utilized not only for design and construction purposes, but also for other tasks such as maintenance management and site surveys. That is, not only can it be used as a design drawing, but other applications such as leaving records on the 3D model and sharing with others can also be considered.

[0014] ​Furthermore, since the work performed on the 3D model can be recorded as a log, if tacit knowledge can be extracted from this data, it can be effectively used for transferring skills from experienced personnel to younger ones. This is expected to lead to front-loading of operations and talent development.

[0015] Focusing on the transmission of tacit knowledge, the challenge lies not only in 3D models but also in 2D data (such as 3D spherical images and planar images), as mentioned above, in how to transmit tacit knowledge between different tasks and among users with varying levels of skill.

[0016] Specifically, because tacit knowledge is qualitative and difficult to quantify, even if an tacit knowledge model is generated from tacit knowledge, it is difficult to secure user trust in the tacit knowledge model, making it difficult to promote its use. For example, if the field of expertise of the tacit knowledge model differs from that of the user, no matter how excellent the tacit knowledge model is, it will have no value to the user. Similarly, if the knowledge level of the tacit knowledge model is lower than the user's knowledge level, it will have no value to the user.

[0017] However, it is also true that tacit knowledge models can give users new perspectives and insights, and by using tacit knowledge models, even inexperienced users have great potential to acquire know-how and skills and apply them to their work.

[0018] Furthermore, there may be cases where it is desirable to add a function to the system, which includes a first server and terminal devices that hold property management information, to display images such as 3D models corresponding to the property management information.

[0019] One way to achieve this is for the first server to acquire the functionality to display property management information and images such as 3D models. However, this would require adding functionality to the first server, leading to increased costs.

[0020] Therefore, in this embodiment, the second server performs processing based on property management information managed by the first server and three-dimensional image information of the property managed by the second server, so that the user can see the three-dimensional image information and the property management information acquired from the first server linked together on a single screen on the terminal device. The property management information includes photographs taken of the property. In addition, the second server can not only display two pieces of information on the terminal device, but can also display tacit knowledge about the property (such as text information) generated based on the photographs and three-dimensional image information, linked together with the three-dimensional image information. As a result, without adding significant functionality to the first server, the terminal device can display tacit knowledge about the property linked together with the three-dimensional image information.

[0021] <About Terminology> A user is someone who uses the text information (or other content such as images) generated by the tacit knowledge model. A data provider is someone who provides the data (such as audio information, text information, operation information, images, 3D data, etc.) that the tacit knowledge model uses for training.

[0022] Tacit knowledge is knowledge based on an individual's experience, intuition, etc. An tacit knowledge model is a model that learns tacit knowledge and outputs answers based on that learned tacit knowledge in response to questions. A model is a mechanism or artificial intelligence (AI) that learns the correspondence between input data and output data and outputs output data for given input data. Note that the output data does not depend on whether or not there is training data.

[0023] A property can be any space where items can be placed, such as a facility or a room within a facility. Items are things that are placed in the property. The types of items placed will vary depending on the function of the facility.

[0024] For example, properties include real estate, factories, construction sites, research facilities, medical facilities, farmland, warehouses, and equipment requiring maintenance. Goods include furniture, materials, equipment, heavy machinery, tools, instruments, materials, cultured products, and food.

[0025] Information concerning an item includes the item's name and the name of the property where the item is located, but it is preferable that the information be tacit knowledge. For example, information concerning an item may include information known only to a specific user. In this embodiment, information concerning an item is described using the term "inspection information." Other information concerning an item may include the item's instruction manual, daily reports of work performed on the item, an item's quotation, or an item's drawing. Inspection may also include investigation, testing, verification, review, checking, etc.

[0026] Three-dimensional image information of an object refers to the image of the object displayed on a screen. While three-dimensional image information of an object is a two-dimensional image projected from the three-dimensional model shape of the object, the user can change their viewpoint. Furthermore, three-dimensional image information of an object may also be a 360-degree spherical image.

[0027] A display screen is a screen that shows, for example, inspection information, captured images, and three-dimensional image information of an item on a single screen. In this embodiment, the screens shown in Figures 12 and 19 are examples.

[0028] The generated information is information generated based on three-dimensional image information and captured images. The generated information may be generated by an implicit knowledge model. In this embodiment, it is described using the terms implicit knowledge comments or text information.

[0029] [First Embodiment] <Example System Configuration> Figure 1 is an overall configuration diagram of an example of an information processing system 100. The information processing system 100 in this embodiment includes a terminal device 10, an image management server 40, and a property information management server 20, which are examples of input / output devices. However, the terminal device 10 may be connected to the image management server 40 or the property information management server 20 as needed, and the information processing system 100 does not necessarily have to include it.

[0030] The image management server 40 (an example of a second server) is one or more information processing devices that can communicate with the terminal device 10 via a communication network N. The image management server 40 manages three-dimensional image information of the object and also has an implicit knowledge model and a large-scale language model. The image management server 40 uses these to return text information containing the implicit knowledge to the user. The image management server 40 may be a web server that returns processing results to the terminal device 10 in response to requests from the terminal device 10. A server is a computer or software that performs the function of providing information and processing results in response to requests from a client.

[0031] The image management server 40 may support cloud computing. Cloud computing refers to a usage model in which network resources are utilized without being aware of specific hardware resources. Cloud computing can take the form of SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service), and any of these forms is acceptable. Therefore, the image management server 40 does not need to be housed in a single chassis or provided as a single unit of equipment. The image management server 40 may have its functions distributed among multiple information processing units, or multiple information processing units may each have all the functions, and the information processing unit being used may be switched through load balancing or other means. Furthermore, the image management server 40 may be a server located in an on-premises environment.

[0032] Furthermore, instead of the image management server 40 having both an implicit knowledge model and a large-scale language model, the image management server 40 may call an API (Application Programming Interface) exposed by an external system and utilize at least one of the implicit knowledge model or the large-scale language model.

[0033] The property information management server 20 (an example of the first server) is one or more information processing devices that can communicate with the terminal device 10 via the communication network N. The property information management server 20 manages property management information. Property management information includes, for example, strings of text, as well as images taken during inspections, etc. The property information management server 20 is a server that allows users to update property management information as needed to manage the progress of construction work on the property, the placement of items, etc.

[0034] The property information management server 20 may be a web server that returns processing results to the terminal device 10 in response to requests from the terminal device 10. The property information management server 20 can communicate with the image management server 40 via the communication network N. The property information management server 20 may also support either cloud computing or on-premises deployment.

[0035] It is preferable that the image management server 40 and the property information management server 20 are linked to a degree that enables single sign-on. The image management server 40 can communicate with the property information management server 20 via an API exposed by the property information management server 20. Alternatively, the image management server 40 and the property information management server 20 may have a business partnership.

[0036] Terminal device 10 is a general-purpose information processing terminal used by users of the information processing system 100. Terminal device 10 runs a web browser and native applications dedicated to the image management server 40 or property information management server 20. When terminal device 10 runs a web browser, terminal device 10 and the image management server 40 or property information management server 20 run a web application. A web application is an application that operates through the cooperation of a program written in a programming language (e.g., JavaScript®) that runs on a web browser and a program on the web server (image management server 40). When a web application is executed, the processing of this embodiment may be performed by the image management server 40 or property information management server 20, or by terminal device 10 that receives the web application.

[0037] An application that cannot be executed unless it is installed on the terminal device 10 is called a native application. In this embodiment as well, the application executed on the terminal device 10 may be either a web application or a native application. In this case as well, the processing in this embodiment may be performed by the image management server 40 or by the terminal device 10 that is running the native application.

[0038] Terminal device 10 can be, for example, a PC (Personal Computer), a smartphone, a PDA (Personal Digital Assistant), or a tablet device. Alternatively, terminal device 10 can be any device capable of running a web browser or native applications. Terminal device 10 may also be an electronic whiteboard, a television receiver, a glasses device, or a wearable device. Furthermore, there may be multiple terminal devices 10.

[0039] Terminal device 10 can communicate with image management server 40 and property information management server 20 via communication network N. Communication network N is constructed using the Internet, LAN (Local Area Network), provider services, etc. Communication network N may include not only wired communication but also mobile communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), and wireless LAN networks. Furthermore, terminal device 10 can communicate using short-range communication technologies such as Bluetooth (registered trademark) and NFC (registered trademark, Near Field Communication).

[0040] In Figure 1, the image management server 40, the property information management server 20, and the terminal device 10 communicate via the communication network N. However, users may also operate the image management server 40 or the property information management server 20 directly from a console.

[0041] <Example Hardware Configuration> Figure 2 is a hardware configuration diagram of an example of the image management server 40, property information management server 20, or terminal device 10 according to this embodiment. The hardware components of the image management server 40 or property information management server 20 are indicated by reference numerals in the 400s. The hardware components of the terminal device 10 are indicated by reference numerals in the 100s.

[0042] The hardware configuration of the terminal device 10 will be described below, but the hardware configuration of the image management server 40 or the property information management server 20 is the same and will therefore not be described.

[0043] The terminal device 10 is built using a computer and, as shown in Figure 2, includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HD (Hard Disk) 104, HDD (Hard Disk Drive) controller 105, display I / F 106, and communication I / F 107.

[0044] Of these components, the CPU 101 controls the overall operation of the terminal device 10. The ROM 102 stores programs used to drive the CPU 101, such as the IPL (Initial Program Loader). The RAM 103 is used as the work area for the CPU 101.

[0045] HD104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to HD10 according to the control of the CPU 101.

[0046] The display I / F 106 is a circuit that displays images on the display 106a. The display 106a is a type of display unit such as a liquid crystal or organic EL (electroluminescence) that displays various information such as cursors, menus, windows, characters, or images. The communication I / F 107 is an interface used for communication with other devices.

[0047] If the terminal device 10 is a glass device, the terminal device 10 may use a circuit that displays an image on a lens or the like, which is a transmissive reflective material, instead of the display I / F 106.

[0048] Communication I / F107 is, for example, a NIC (Network Interface Card) that supports TCP (Transmission Control Protocol) / IP (Internet Protocol).

[0049] Furthermore, the terminal device 10 is equipped with a sensor I / F 108, an audio input / output I / F 109, an input I / F 110, a media I / F 111, and a DVD-RW (Digital Versatile Disk Rewritable) drive 112.

[0050] The sensor interface 108 is an interface for receiving detection information from various sensors. The sound input / output interface 109 is a circuit that processes the input and output of sound signals between the speaker 109a and the microphone 109b according to the control of the CPU 101. The input interface 110 is an interface for connecting a predetermined input means to the terminal device 10.

[0051] Keyboard 110a is a type of input device equipped with multiple keys for inputting characters, numbers, various instructions, etc. Mouse 110b is a type of input device used for selecting and executing various instructions, selecting processing targets, moving the cursor, and operating on the display screen, etc.

[0052] The media interface 111 controls the reading or writing (storage) of data to or from a recording medium 111a such as flash memory. The DVD-RW drive 112 controls the reading or writing of various types of data to or from a DVD-RW 112a, which is an example of a removable recording medium. Note that the DVD-RW may be a DVD-R or the like. The DVD-RW drive 112 may also be a Blu-ray drive that controls the reading or writing of various types of data to or from a Blu-ray Disc (registered trademark).

[0053] Furthermore, the terminal device 10 is equipped with a bus line 113. The bus line 113 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 101.

[0054] Furthermore, recording media such as HDs and CD-ROMs on which the above programs are stored can be provided domestically or internationally as program products. The terminal device 10 realizes the information processing method according to the present invention by executing, for example, the program according to the present invention.

[0055] <About the features> Figure 3 is a functional configuration diagram illustrating an example of the functions of the image management server 40, the property information management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment.

[0056] <<Terminal device>> As shown in Figure 3, the terminal device 10 includes a transmitting / receiving unit 11, an input receiving unit 12, a display control unit 13, an audio control unit 14, a conversion unit 15, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 101 that follow a program deployed from the HD 104 onto the RAM 103. The terminal device 10 also has a storage unit 1000 constructed from at least one of the RAM 103 and HD 104 shown in Figure 2.

[0057] The transmitting / receiving unit 11 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 101 shown in Figure 2, as well as the communication I / F 107, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0058] The input receiving unit 12 is an example of an input receiving means and is mainly implemented by commands from the CPU 101 shown in Figure 2, as well as the input I / F 110 and the sound input / output I / F 109, and accepts various inputs from the user using the microphone 109b, keyboard 110a and mouse 110b.

[0059] The display control unit 13 is an example of a display control means and output means, and is implemented by commands from the CPU 101 shown in Figure 2 and the display I / F 106, and causes various images and screens to be displayed on the display 106a, which is an example of a display unit. If the terminal device 10 is a glasses device, the display control unit 13 causes a virtual image to be displayed on a transmissive reflective member such as a lens instead of the display I / F 106.

[0060] The audio control unit 14 is an example of an audio control means and output means, and is implemented by commands from the CPU 101 shown in Figure 2 and the audio input / output I / F 109, causing the speaker 109a, which is an example of an audio playback unit, to play sound.

[0061] The conversion unit 15 is an example of a processing means, and is implemented by instructions from the CPU 101 shown in Figure 2. It performs processing to convert text information into audio information and processing to convert audio information into text information.

[0062] The storage / reading unit 19 is an example of a storage control means and is executed by instructions from the CPU 101 shown in Figure 2, as well as by the HD 104, media I / F 111, and DVD-RW drive 112. It performs processing such as storing various data in the storage unit 1000, recording media 111a, and DVD-RW 112a, and reading various data from the storage unit 1000, recording media 111a, and DVD-RW 112a.

[0063] <Image management server functional configuration> The image management server 40 includes a transmission / reception unit 41, a screen generation unit 42, a determination unit 43, a identification unit 44, a text information generation unit 45, an update unit 46, a processing unit 47, and a storage / reading unit 49. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 following a program deployed from the HD 404 onto the RAM 403. The image management server 40 also has a storage unit 4000 constructed from the HD 404 shown in Figure 2. The storage unit 4000 is an example of a storage means.

[0064] In Figure 3, all functions are installed in a single image management server 40, but the image management server 40 may also be configured to distribute each function across multiple computers.

[0065] The transmitting / receiving unit 41 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0066] The screen generation unit 42 is an example of a screen generation means and is implemented by instructions from the CPU 401 shown in Figure 2, generating various screens. When the terminal device 10 runs a web application, screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc. Therefore, screen information may also be referred to as a web application. When the terminal device 10 runs a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0067] The decision unit 43 is an example of a decision-making mechanism, and is implemented by instructions from the CPU 401 shown in Figure 2, and performs various decisions described later.

[0068] The identification unit 44 is an example of an identification means, and is implemented by instructions from the CPU 401 shown in Figure 2, and identifies the target image.

[0069] The text information generation unit 45 is an example of a text information generation means, and is implemented by instructions from the CPU 401 shown in Figure 2. It obtains implicit knowledge comments from an implicit knowledge model and generates text information based on a large-scale language model 4005.

[0070] The update unit 46 is an example of an update means, and is implemented by instructions from the CPU 401 shown in Figure 2, and performs an update of the implicit knowledge model, which will be described later.

[0071] The processing unit 47 performs a process to associate the three-dimensional image information of an item with the inspection information or generated information based on the inspection information or captured images transmitted by the property information management server 20 and the three-dimensional image information of the item. Furthermore, the processing unit 47 also performs a process to associate the three-dimensional image information with the captured images. The processing based on the inspection information or captured images and the three-dimensional image information includes a process to display both on one screen, or a process to obtain an implicit knowledge comment, which is an example of text information, from the implicit knowledge model 4004 using the inspection information or captured images and the three-dimensional image information. The processing unit 47 requests processing from the screen generation unit 42 or the text information generation unit 45, etc., depending on the content of the processing. The storage / reading unit 49 is an example of a storage control means and is executed by instructions from the CPU 401 shown in Figure 2, as well as by the HD 404, media I / F 411, and DVD-RW drive 412. It performs processing such as storing various data in the storage unit 4000, recording media 411a, and DVD-RW 412a, and reading various data from the storage unit 4000, recording media 411a, and DVD-RW 412a. The storage unit 4000, recording media 411a, and DVD-RW 412a are examples of storage means.

[0072] The memory unit 4000 contains an image information management DB 4001, a model shape management DB 4002, a caption model 4003, an implicit knowledge model 4004, and a large-scale language model 4005.

[0073] Image information management DB 4001 manages three-dimensional image information related to the property. This three-dimensional image information is information that visually represents the items (also called models) placed on the property. Model shape management DB 4002 manages three-dimensional model shape information of the items placed on the property. Image management server 40 can generate three-dimensional image information related to the property from the three-dimensional model shape information. Three-dimensional model shape information is information that depicts the item in three dimensions, such as a three-dimensional model of the item or a three-dimensional point cloud. Three-dimensional model shape information may be composed of polygons or CAD. In addition, it is preferable that image information management DB 4001 or model shape management DB 4002 store wide-field images such as 360-degree images of the property. Furthermore, inspection information and captured images transferred from property information management DB 2001 are stored in image information management DB 4001.

[0074] Caption model 4003 is a model that is generated by performing a training process using image and caption comment combinations as training data, and enables a computer to function to output caption comments based on images. Caption comments are explicit knowledge and are used as a term in contrast to tacit knowledge. Here, caption comments are text data, such as comments expressed in audio or text that describe an image. Caption comments relating to objects or articles are associated with the identification information of those objects or articles.

[0075] The tacit knowledge model 4004 is generated by performing a training process using the correspondence between three-dimensional image information and captured images and comments (comments other than caption comments) for this three-dimensional image information and captured images as training data. It is a model that makes a computer function to output tacit knowledge comments based on images. The tacit knowledge model 4004 may also learn the correspondence between information other than comments and three-dimensional image information and captured images, as described below. Correspondence between three-dimensional image information and captured images and inspection information Correspondence between 3D image information and captured images, and comments and inspection information. Here, tacit knowledge comments are text data, and are comments expressed in audio or text, excluding caption comments, that is, comments relating to content not represented in the image.

[0076] The large-scale language model 4005 is a computer language model that is generated by performing a training process using a vast amount of unlabeled text as training data and consists of an artificial neural network with many parameters. The large-scale language model 4005 can capture much of the syntax and meaning of human language by being sufficiently trained with context-learning techniques such as next sentence prediction, which understands context by determining whether sentence 1 and sentence 2 are consecutive, and the masked language model, which understands context by masking words in a sentence and predicting the masked words from the words before and after them.

[0077] <<Image Information Management Table>> Figure 4 is a conceptual diagram showing an example of an image information management table according to this embodiment. The storage unit 4000 has an image information management DB 4001 constructed, which is composed of the image information management table shown in Figure 4.

[0078] This image information management table manages model identification information, location information, item information, inspection information, 3D image information, and captured images, linked to the item identification information. An item can be any space where items can be placed, such as a facility or a room within a facility. The types of items placed will vary depending on the function of the facility.

[0079] Property identification information is identification information that uniquely identifies a property.

[0080] Model identification information is identification information that identifies items placed on a property. The property management information is associated with the three-dimensional model shape in the model shape management DB4002 using this model identification information. This model identification information is the same as the model identification information in the property information management server 20, but it may also be associated with the model identification information in the property information management server 20 using a conversion table.

[0081] • Location information is information that indicates the position of an object in a virtual space within the generated property using three-dimensional coordinates (XYZ). XYZ can be the centroid of the object, or the coordinates of the vertices of the circumscribing cube.

[0082] • Item information refers to the names of the items placed on the property. In Figure 4, item names are shown assuming a construction site or building site, but item names will vary depending on the property. Also, in Figure 4, properties and items are associated, but items can be registered independently of properties.

[0083] Inspection information is information about inspections performed by users on items within a property. Inspection information varies depending on the item and the purpose of the inspection, but may include, for example, whether there are any abnormalities in the item, dimensions, inspection date and time, and inspector. Inspection information is obtained from the property information management server 20. The property information management server 20 holds master inspection information, and the image management server 40 obtains and stores inspection information from the property information management server 20. This is because item inspections are performed on the property information management server 20. For this reason, inspection information in the image information management table may only exist for some items, or even if inspection information exists, it may not be the most up-to-date inspection information.

[0084] In addition, the inspection information may also be photographic information of the inspected items. In this case, the update unit 46 analyzes the photographic information to extract features such as the items and the condition of any damage, and converts them into inspection information represented as a string of characters.

[0085] In addition to inspection information, instruction manuals, daily reports, quotations, drawings, etc., may also be registered in the image information management table.

[0086] Three-dimensional image information refers to information that visually represents an object, such as three-dimensional model shape information or a 360-degree image. In Figure 4, three-dimensional image information is placed in the image information management table, but the three-dimensional image information may also be stored in the model shape management DB4002.

[0087] Images 1 and 2 are two-dimensional images obtained from the property information management server 20 along with inspection information. The property information management server 20 holds the master images. The images are, for example, two-dimensional images of items taken by inspectors during inspection. Therefore, if inspection information is stored in the image information management DB 4001, the images are often also stored there. There are multiple images, such as Images 1 and 2, because items may be photographed each time an inspection is performed. There may be more than three images. Images 1 and 2 have a date and time of capture.

[0088] <Functional Configuration of Property Information Management Server> Let's return to Figure 3 for explanation. The property information management server 20 has a transmitting / receiving unit 21, a screen generation unit 22, and a storage / reading unit 29. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 following a program deployed from the HD 404 onto the RAM 403. The property information management server 20 also has a storage unit 2000 constructed from the HD 404 shown in Figure 2. The storage unit 2000 is an example of a storage means.

[0089] In Figure 3, all functions are installed on a single property information management server 20, but the property information management server 20 may also be configured to distribute each function across multiple computers.

[0090] The transmitting / receiving unit 21 is an example of a transmitting or receiving unit, and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0091] The screen generation unit 22 is an example of a screen generation means and is implemented by instructions from the CPU 401 shown in Figure 2, generating various screens. When the terminal device 10 runs a web application, screen information is created using HTML, XML, CSS, and JavaScript (registered trademark), etc. Therefore, screen information may also be referred to as a web application. When the terminal device 10 runs a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0092] The storage / reading unit 29 is an example of a storage control means and is executed by instructions from the CPU 401 shown in Figure 2, as well as by the HD 404, media I / F 411, and DVD-RW drive 412. It performs processing such as storing various data in the storage unit 2000, recording media 411a, and DVD-RW 412a, and reading various data from the storage unit 2000, recording media 411a, and DVD-RW 412a. The storage unit 2000, recording media 411a, and DVD-RW 412a are examples of storage means.

[0093] <<Property Management Information Management Table>> Figure 5 is a conceptual diagram showing an example of a property management information management table according to this embodiment. The storage unit 2000 has a property information management DB 2001 constructed, which is composed of the property management information management table shown in Figure 5.

[0094] This property management information management table manages model identification information, location information, item information, inspection information, and one or more photographed images, associated with the property identification information. These are sometimes referred to as property management information. The contents of these are the same as those in the image information management table. The property management information management table does not store three-dimensional image information. However, inspection information and photographed images are recorded for all items that have been inspected at least once. Also, the inspection information and photographed images in the property information management server 20 are inspection information directly registered by the user, so they serve as the master (source) for the inspection information in the image management server 40. Photographed images may be photographs of multiple items, or images other than photographs, such as layout diagrams.

[0095] Alternatively, instead of the property information management DB2001 consisting of the property management information management table described above, a property information management DB2001 consisting of a property management information management table that manages location information, item information, inspection information, and captured images, without associating them with property identification information, may be used.

[0096] <An example of model updating and text information generation> Refer to Figures 6 and 7 to explain how to update the model and generate text information. In Figures 6 and 7, check information is not used for updating the model and generating text information, but similar learning can be achieved by replacing or adding check information to dialogue Q1, etc. below.

[0097] Figure 6 is an explanatory diagram of what the terminal device 10 displays during text information generation processing and model update processing. Figure 6(a) is an explanatory diagram of the model update processing. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes the target image 1100 and text 1200.

[0098] The input receiving unit 12 of the terminal device 10 receives audio information from the microphone 109b, which represents the dialogue Q1, A1, Q2, and A2 between data providers M1 and M2, as input information to be entered by the data provider in response to the displayed screen 900. It is preferable that data providers M1 and M2 have extensive knowledge, including tacit knowledge, about their work. Through such a dialogue between data providers M1 and M2, the tacit knowledge model 4004 is updated, allowing the user to obtain useful tacit knowledge comments.

[0099] The identification unit 44 identifies the target image 1100, which is the portion of the display screen 900 excluding the text 1200.

[0100] The judgment unit 43 then uses the target image 1100 to determine the degree of relevance between the caption comments obtained from the caption model 4003 and the dialogues Q1, A1, Q2, and A2.

[0101] The update unit 46 updates the tacit knowledge model 4004 by using comments from the dialogues Q1, A1, Q2, and A2 that are judged to have a low degree of relevance as tacit knowledge comments, along with the target image 1100, as training data, and updates the caption model 4003 by using comments that are judged to have a high degree of relevance as caption comments, along with the target image 1100, as training data.

[0102] Therefore, the tacit knowledge model 4004 learns the correspondence between the target image 1100 and the dialogues Q1, A1, Q2, and A2. The target image 1100 has features extracted using several feature extraction models suitable for images, such as a CNN (Convolutional Neural Network). These features represent, for example, what is depicted and where, or the tasks being performed within the image. Thus, the tacit knowledge model 4004 can learn the correspondence between the image features and the dialogues Q1, A1, Q2, and A2.

[0103] Figure 6(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes an image 1110 and text 1210.

[0104] The input receiving unit 12 of the terminal device 10 receives audio information indicating questions Q11 and Q12 from user M3 via the microphone 109b as input information to be entered by the user on the displayed screen 900.

[0105] The identification unit 44 identifies the image 1110 that does not contain the text 1210 as the target image.

[0106] The text information generation unit 45 uses image 1110 to acquire tacit knowledge comments based on the tacit knowledge model 4004. The tacit knowledge model 4004 extracts features from image 1110 and determines that the features of image 1110 in Figure 6(b) are similar to those of image 1110 at the time of update, and can identify dialogues Q1, A1, Q2, and A2 related to image 1110. Dialogues Q1, A1, Q2, and A2 become tacit knowledge comments.

[0107] Furthermore, the text information generation unit 45 uses tacit knowledge comments (dialogue Q1, A1, Q2, A2) and questions Q11, Q12, etc., to generate text information related to the answers A11 and A12 to questions Q11 and Q12, respectively, based on the large-scale language model 4005.

[0108] The display control unit 13 of the terminal device 10 displays the text information related to the answers A11 and A12 received from the image management server 40 on the display 106a.

[0109] Figure 7 is another explanatory diagram displayed by the terminal device 10 during the model update process and text information generation process according to this embodiment. Figure 7 illustrates a case in which no question text is used during model update and text information generation.

[0110] Figure 7(a) is an explanatory diagram of the model update process. Figure 7(a) shows an example in which the tacit knowledge model 4004 is updated with audio information and partial images from a single data provider, rather than a conversation between data providers.

[0111] The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes a first image 1100A and a second image 1100B.

[0112] The input receiving unit 12 of the terminal device 10 receives character information from the keyboard 110a, which represents comments C1 to C4 from data provider M4, as input information to be entered by the data provider in response to the displayed screen 900.

[0113] Furthermore, the input receiving unit 12 receives operation information from the mouse 110b, which indicates an operation by the data provider M4 to identify a partial image 1100B1 in the second image 1100B, as input information to be entered by the data provider M4 into the displayed screen 900.

[0114] The identification unit 44 may identify the partial image 1100B1 as the target image, or it may identify the first image 1100A or the second image 1100B as the target image.

[0115] The judgment unit 43 then uses the target image to determine the degree of relevance between the caption comments obtained from the caption model 4003 and comments C1 to C4.

[0116] The update unit 46 updates the tacit knowledge model 4004 by using comments C1 to C4 that are judged to have low relevance as tacit knowledge comments, along with partial image 1100B1, etc., as training data, and updates the caption model 4003 by using comments that are judged to have high relevance as caption comments, along with partial image 1100B1, as training data.

[0117] Therefore, the tacit knowledge model 4004 learns the correspondence between the partial image 1100B1 and the comments C1-C4. Features are extracted from the partial image 1100B1 using several feature extraction models suitable for images, such as CNNs. These features represent, for example, what is depicted and where, or the content of the work being performed. Thus, the tacit knowledge model 4004 can learn the correspondence between the features of the image and the comments C1-C4.

[0118] Figure 7(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes the image 1110.

[0119] User M5 does not input anything to the displayed screen 900, and the input reception unit 12 does not accept any input information from the user to the displayed screen 900. The identification unit 44 identifies the entire displayed screen 900, which is image 1110, as the target image.

[0120] When user M5 performs an operation to identify a partial image 1100B1 on the display screen 900, the input receiving unit 12 receives operation information from the mouse 110b indicating the operation to identify the partial image as input information. In this case, the identification unit 44 identifies the partial image on the display screen 900 as the target image according to the operation information.

[0121] The text information generation unit 45 uses the partial image 1100B1 to acquire tacit knowledge comments based on the tacit knowledge model 4004. The tacit knowledge model 4004 determines that the features of image 1110B1 in Figure 7(b) are similar to those of image 1110B1 at the time of update, and can identify comments C1 to C4 related to image 1110B1. The tacit knowledge model 4004 extracts comments C1 to C4 as tacit knowledge comments.

[0122] The text information generation unit 45 generates text information related to comments C11 to C14 based on the large-scale language model 4005, using tacit knowledge comments and the like. The text information generation unit 45 may also generate text information using preset standard questions if no question text is entered, rather than using a method that does not use any questions at all.

[0123] The display control unit 13 of the terminal device 10 displays the text information related to comments C11 to C14 received from the image management server 40 on the display 106a.

[0124] <Action or process> Next, we will explain how to display inspection information, three-dimensional image information, and captured images on a single screen as an example of processing based on these. In other words, the tacit knowledge model 4004 is not used.

[0125] Figure 8 is a sequence diagram showing an example of a process that generates screen information by arranging inspection information, three-dimensional image information, and captured images.

[0126] S1: The user enters a login operation into terminal device 10. This login is for the property information management server 20. The input reception unit 12 of terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful.

[0127] Users log in to the property information management server 20 first, and then log in to the image management server 40. However, they may also log in to the image management server 40 first, and then log in to the property information management server 20.

[0128] S2: Upon successful login, the transmitting / receiving unit 11 of the terminal device 10 sends a request for the property selection screen 200 to the property information management server 20.

[0129] S3: The transmitting / receiving unit 21 of the property information management server 20 receives a request for the property selection screen 200. The screen generation unit 22 generates the property selection screen 200, and the transmitting / receiving unit 21 transmits the screen information of the property selection screen 200 to the terminal device 10.

[0130] S4: The transmitting / receiving unit 11 of the terminal device 10 receives screen information from the property selection screen 200. The display control unit 13 displays the property selection screen 200 (see Figure 9). The user enters property identification information (e.g., V0001) on the displayed property selection screen 200. The input receiving unit 12 of the terminal device 10 receives the property identification information.

[0131] S5: The transmitting / receiving unit 11 of the terminal device 10 sends a request for property management information to the property information management server 20, specifying the property identification information.

[0132] S6: The transmitting / receiving unit 21 of the property information management server 20 receives a request for property management information, and the storage / reading unit 29 searches the property information management DB 2001 using the property identification information. The property management information is information about the property associated with the property identification information. The screen generation unit 22 of the property information management server 20 generates a property management screen 210 that displays the property management information, and the transmitting / receiving unit 21 transmits the screen information of the property management screen 210 to the terminal device 10.

[0133] Furthermore, the transmitting / receiving unit 21 sends an image request program to the terminal device 10 in response to a request for property management information, so that the terminal device 10 can acquire three-dimensional image information. The image request program is, for example, a web application. This web application is installed on the property information management server 20 by the operator of the image management server 40 with the permission of the operator of the property information management server 20. Alternatively, a URL containing the image request program may be sent to the terminal device 10. Since the web application acquires three-dimensional image information from the image management server 40, it has the function of connecting to the image management server 40 and requesting and displaying three-dimensional image information.

[0134] S7: The transmitting / receiving unit 11 of the terminal device 10 receives screen information and an image request program from the property management screen 210. The display control unit 13 displays the property management screen 210 (see Figure 10). This displays the property management information. The user requests three-dimensional image information of the property (by pressing the image acquisition button 213) based on the displayed property management information. The input receiving unit 12 of the terminal device 10 receives the request for three-dimensional image information of the property. Three-dimensional image information of the property is three-dimensional image information of items placed in the property, which has been generated as a virtual space. The items are composed of 3D model shape information. Since the property has already been identified, the request for three-dimensional image information of the property may be sent to the image management server 40 without any operation by the user.

[0135] Here, the property management screen 210 includes a first display area 214 that displays information about the item obtained from the property information management server 20, and a second display area 215 that displays three-dimensional image information of the item obtained from the image management server 40. At step S7, the property management information is displayed in the first display area 214, but nothing is displayed in the second display area 215.

[0136] S8: If the user is not logged in to the image management server 40, the user enters a login operation into the terminal device 10. This login operation is directed to the image management server 40. The input reception unit 12 of the terminal device 10 accepts the login operation. Any existing login method is acceptable. Here, we assume that the login was successful. Alternatively, the user's login operation may be omitted by using, for example, single sign-on.

[0137] S9: The terminal device 10 executes an image request program to request three-dimensional image information. The transmitting / receiving unit 11 then sends a request for three-dimensional image information of the property selected by the user to the image management server 40, specifying the property identification information of the property. The transmitting / receiving unit 11 may also send the URL of the property information management server 20 to the image management server 40 so that the terminal device 10 can be redirected to the property information management server 20. Three-dimensional image information of a property is an image of an object placed in a property formed as a virtual space. Since the object is composed of 3D model shape information, the terminal device 10 generates a two-dimensional image by projecting the three-dimensional model shape of the object onto two dimensions. The user can view any object while changing their viewpoint. The transmitting / receiving unit 11 may also have the image management server 40 receive property management information obtained from the property information management server 20. For example, the image request program receives property management information from a web application connected to the property information management server 20 as a URL parameter.

[0138] S10: The transmitting / receiving unit 41 of the image management server 40 receives a request for three-dimensional image information of an object. The storage / reading unit 49 searches the image information management DB 4001 using the object identification information and obtains three-dimensional image information for each item. The processing unit 47 requests the screen generation unit 42 to generate a screen containing the three-dimensional image information of the object and the captured image. The screen generation unit 42 generates a screen corresponding to a second display area 215 in which the three-dimensional image information and the captured image of each item are arranged on one screen. The screen generation unit 42 also generates a screen corresponding to a first display area 214 in which the captured image is arranged. The transmitting / receiving unit 41 transmits the screen information of the screens corresponding to the first display area 214 and the second display area 215 to the terminal device 10. The three-dimensional image information of each item included in this screen information is three-dimensional image information in which all items included in the object are placed on the object, and the user can arbitrarily change the viewpoint.

[0139] S11: The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215, and the display control unit 13 displays the property display screen 220 including the first display area 214 and the second display area 215 (see Figure 11). At step S11, only the three-dimensional image information of each item is displayed in the second display area 215, but the first display area 214 displays, for example, a list of items placed in the property (generated from property management information), so that the item list and the three-dimensional image information of the property are displayed on one screen. The item list is included in the property management information. Next, the user identifies an arbitrary item from the three-dimensional image information of the property. Once the user has identified an item, they can request inspection information and captured images of the item. The input receiving unit 12 of the terminal device 10 accepts the operation to identify an item. The item may be identified, for example, by the coordinates clicked by the user, or model identification information may be identified by these coordinates.

[0140] S12: When the user presses the information display button 225, the transmitting / receiving unit 11 of the terminal device 10 transmits information identifying the item to the image management server 40.

[0141] S13: The transmitting / receiving unit 41 of the image management server 40 receives information that identifies an item. The storage / reading unit 49 identifies the model identification information of the item with the closest location information from the image information management DB 4001 based on the information that identifies the item (in this case, location information). The transmitting / receiving unit 41 transmits the model identification information to the terminal device 10. Note that if the information that identifies the item in step S12 is model identification information, searching the image information management DB 4001 is unnecessary. The image management server 40 transmits the model identification information in order to obtain the latest inspection information from the property information management server 20.

[0142] S14: The transmitting / receiving unit 11 of the terminal device 10 receives the model identification information. For example, the image management server 40 notifies the terminal device 10 of the URL of the property information management server 20 and redirects it. As a result, the transmitting / receiving unit 11 of the terminal device 10 sends the property information management server 20 a request for inspection information and captured images of the item, specifying the model identification information.

[0143] S15: The transmitting / receiving unit 21 of the property information management server 20 receives requests for inspection information and captured images of an item, and the storage / reading unit 29 searches the property information management DB 2001 using the model identification information to obtain the inspection information and captured images. The transmitting / receiving unit 21 transmits the inspection information and captured images to the terminal device 10. If multiple captured images are already registered in the property information management DB 2001, the storage / reading unit 29 may obtain the latest captured image, obtain N new captured images without duplication, or obtain all captured images.

[0144] S16: When the transmitting / receiving unit 11 of the terminal device 10 receives inspection information and captured images, it specifies the model identification information and sends the inspection information and captured images to the image management server 40. When the transmitting / receiving unit 41 of the image management server 40 receives the inspection information and captured images as a response to the request in step S13, the storage / reading unit 49 identifies the item using the model identification information and saves (or overwrites) the latest inspection information and captured images in the image information management DB 4001. In other words, it saves inspection information and captured images that do not yet exist in the image information management DB 4001.

[0145] S17: Next, the processing unit 47 associates the three-dimensional image information of the item corresponding to the model identification information with the latest inspection information and the captured image, and requests the screen generation unit 42 to display them. As processing based on the information about the item and the three-dimensional image information, the screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the inspection information, the captured image, and the three-dimensional image information of the item in association. The screen generation unit 42 may also perform an update process to add only the inspection information to the screen corresponding to the second display area 215. The screen generation unit 42 also generates a screen that displays the captured image in the first display area 214. The transmitting / receiving unit 41 transmits the screen information of the screen corresponding to the second display area 215 to the terminal device 10.

[0146] S18: The transmitting / receiving unit 11 of the terminal device 10 receives screen information for the screen corresponding to the second display area 215, and the display control unit 13 displays the inspection image screen 230 including the first display area 214 and the second display area 215 (see Figure 12). In step S18, the first display area 214 displays the item list and the captured image, and the second display area 215 displays the three-dimensional image information of the item, inspection information corresponding to the three-dimensional image information, and the captured image.

[0147] <Screen example> Figure 9 shows an example of a property selection screen 200 for entering property identification information. The property selection screen 200 has a property identification information input field 201 and a search button 202. When the user enters the property identification information in the property identification information input field 201 and presses the search button 202, the list of room numbers shown in Figure 10 is displayed on the property management screen 210.

[0148] Figure 10 shows an example of a property management screen 210. The property management screen 210 includes a first display area 214 that displays information about items obtained from the property information management server 20, and a second display area 215 that displays three-dimensional image information of items obtained from the image management server 40. The first display area 214 is the area other than the second display area 215. The first display area 214 includes a list 211 of room numbers belonging to the property identified by the property identification information. The list 211 of room numbers corresponds to property management information, and includes items contained in each property. Depending on the property, the room number may not be displayed, and the user may transition from the property selection screen 200 in Figure 9 to Figure 11 to display the three-dimensional image information of the property. The user selects the room number for which they want to display the three-dimensional image information using the mouse cursor 212. When the user presses the image acquisition button 213, the property display screen 220 is displayed.

[0149] The second display area 215 is an area other than the first display area 214, and its display may be implemented by a program on a web application, such as an iframe.

[0150] Figure 11 shows an example of a property display screen 220. The property display screen 220 includes a first display area 214 and a second display area 215. The first display area 214 of the property display screen 220 displays an item list 221 and a photographed image 251. The items in this room are a table, a cylindrical monument, a door, and a storage box. Also, the photographed image 251 in Figure 11 is a layout diagram and not a photograph taken during inspection. Thus, a layout diagram may be displayed instead of a photographed image at a stage before a property is specified. The image management server 40 can send a layout diagram that substitutes a photographed image to the terminal device 10.

[0151] The second display area 215 of the property display screen 220 displays three-dimensional image information 222 and a photographed image 252 (layout diagram) of these items. The photographed image 252 in the second display area 215 may be the same as the photographed image 251 in the first display area 214. The reason why the same photographed images 251 and 252 are displayed in both the first display area 214 and the second display area 215 is to ensure that the photographed image 252 is displayed even when the second display area 215 is displayed in full screen.

[0152] In addition, the second display area 215 may display an image related to a property other than the layout diagram.

[0153] Next, the user can use the mouse cursor 212 to select the item(s) for which they want to display inspection information and images other than layout diagrams from the three-dimensional image information. This determines the coordinates of the item(s) as information to identify the item(s). Alternatively, the user may select an item(s) from the item list 221. In this case, model identification information is identified at the time of selection. When the user presses the information display button 225, the inspection image screen 230 is displayed. The information display button 225 is used to display inspection information and images, as well as text information generated based on tacit knowledge comments, as will be described later. Furthermore, when the user presses the information update button 226, the tacit knowledge model 4004 is updated.

[0154] In Figure 11, the area (floor area) of 224 is displayed as information about the property. The area (floor area) of 224 may be a measured value or may be included in the property management information.

[0155] As shown in Figure 11, the terminal device 10 was able to display property management information managed by the property information management server 20, three-dimensional image information 222 of the property managed by the image management server 40, and captured images 251 and 252 on a single screen. Users could view the property management information while checking the three-dimensional image information 222 and captured images 251 and 252 of the property.

[0156] Figure 12 shows an example of an inspection image screen 230. The inspection image screen 230 includes a first display area 214 and a second display area 215. In Figure 12, as an example of an item, three-dimensional image information 223 of a table selected by the user is displayed in the second display area 215. The second display area 215 also displays inspection information 232 associated with the table selected by the user, and a captured image 254. An example of inspection information 232 is "Table: Size ○mm*○mm*○mm". The captured image 254 in the second display area 215 is obtained from the property information management server 20. The terminal device 10 can display multiple captured images in the second display area 215, including past captured images, in addition to the captured image 254.

[0157] Note that the captured image 253 for the first display area 214 may be the same image as the captured image 254 for the second display area 215.

[0158] In this way, the terminal device 10 can display the three-dimensional image information 223 of the item, the inspection information 232 of the item on which the three-dimensional image information 223 is displayed, and the captured image 254 of the item on a single screen.

[0159] From the state shown in Figure 12, the user moves the mouse cursor 212 to the cylindrical monument on the item list 221 and presses it. The image management server 40 receives a message that the cylindrical monument has been pressed, and the screen generation unit 42 generates a screen corresponding to the second display area 215, which displays the inspection information of the cylindrical monument, the three-dimensional image information of the item, and N captured images in association. The transmitting / receiving unit 41 transmits the screen information of the screen corresponding to the second display area 215 to the terminal device 10.

[0160] Figure 13 illustrates an example of switching between three-dimensional image information and inspection information of an item displayed in the second display area 215. In Figure 13, an item called a cylindrical monument is selected. Therefore, the three-dimensional image information 227 and inspection information 228 of the cylindrical monument are displayed in the second display area 215. Note that the captured image 255 in the first display area 214 may be the same image as the latest captured image 256 displayed in the second display area 215.

[0161] As shown in Figure 12, when the three-dimensional image information 223 of the first item (table) is displayed, the information (inspection information 232) associated with the three-dimensional image information of the first item is displayed. In this state, when the display changes from the three-dimensional image information of the first item to the three-dimensional image information 227 of the second item (cylindrical monument), the display changes to the information (inspection information 228) associated with the three-dimensional image information 227 of the second item and the captured images 256 and 257.

[0162] Furthermore, as shown in Figure 13, the terminal device 10 can display multiple captured images 256 and 257 in the second display area 215. The second display area 215 displays two captured images 256 and 257 of the cylindrical monument taken from different viewpoints. Captured image 257 is, for example, an image taken in the past than the most recent captured image 256. The user can specify the date and time of capture to display multiple captured images 256 and 257. The second display area 215 may display not only two but also three or more captured images. The user can enlarge the selected captured image.

[0163] <Example of an image management server retrieving inspection information from a property information management server> In Figure 8, the terminal device 10 acquires inspection information and captured images from the property information management server 20, and the image management server 40 acquires them from the terminal device 10. However, it is also possible for the image management server 40 to directly acquire inspection information and captured images from the property information management server 20.

[0164] Figure 14 is a sequence diagram showing an example of a process that generates screen information based on inspection information, three-dimensional image information, and captured images. Note that Figure 14 may mainly explain the differences from Figure 8. First, the processes in steps S11 to S22 can be the same as in Figure 8.

[0165] S21: The transmitting / receiving unit 41 of the image management server 40 receives information that identifies an item. The storage / reading unit 49 identifies the model identification information of the item with the closest location information based on the information that identifies the item (in this case, location information) from the image information management DB 4001. The processing unit 47 requests inspection information from the transmitting / receiving unit 41. The transmitting / receiving unit 41 calls the API of the property information management server 20 and sends the inspection information and captured image request to the property information management server 20, specifying the model identification information. Note that if the information that identifies the item is model identification information, searching the image information management DB 4001 is unnecessary.

[0166] S22: The transmitting / receiving unit 21 of the property information management server 20 receives a request for inspection information and captured images. The storage / reading unit 29 searches the property information management DB 2001 using the model identification information and obtains the inspection information and captured images. The transmitting / receiving unit 21 sends the inspection information and captured images to the image management server 40. The transmitting / receiving unit 41 obtains the inspection information and captured images from the property information management server 20 as a response to the request in step S21.

[0167] The subsequent processing can be the same as in Figure 8. With the processing shown in Figure 14, the processing required for the terminal device 10 to change the connection destination can be reduced, thus shortening the time until the inspection image screen 230 is displayed.

[0168] <Main effects, etc.> In this embodiment, the image management server 40 performs processing based on inspection information and three-dimensional image information, so the terminal device 10 can display the three-dimensional image information and the second display area 215 of the inspection image screen 230, which includes inspection information and captured images corresponding to the three-dimensional image information. The image management server 40 can perform processing based on the inspection information and captured images managed by the property information management server 20, and the three-dimensional image information managed by the image management server 40, without adding processing functions to the property information management server 20.

[0169] Furthermore, the property information management server 20 may perform some processing based on the inspection information and captured images managed by the property information management server 20, and the three-dimensional image information managed by the image management server 40. Even in this case, the processing load on the property information management server 20 is reduced compared to performing all processing based on the inspection information and captured images managed by the property information management server 20, and the three-dimensional image information managed by the image management server 40, on the property information management server 20 side.

[0170] Here, the three-dimensional image information and the inspection information corresponding to the three-dimensional image information may be displayed in a way that does not overlap with each other, or they may be displayed in a way that overlaps with each other. Furthermore, the first display area 214 and the second display area 215 may be displayed so as not to overlap with each other, or they may be displayed so as to overlap with each other. Furthermore, the first display area 214 and the second display area 215 may each be divided into multiple sections for display, and the divided sections may be displayed in a mixed manner.

[0171] [Second Embodiment] In this embodiment, we will describe an image management server 40 that acquires tacit knowledge comments from a tacit knowledge model using inspection information, three-dimensional image information, and captured images, and generates text information based on the tacit knowledge comments.

[0172] In this embodiment, we will explain the process assuming that the hardware configuration diagram in Figure 2 and the functional block diagram in Figure 3, as described in the above embodiment, can be used as reference.

[0173] <Action or process> <<Learning Phase (Model Update)>> First, referring to Figure 15, we will explain the model update process in which the tacit knowledge model 4004 learns data. Figure 15 is a sequence diagram showing an example of the model update process. In explaining Figure 15, we will mainly explain the differences from Figure 8. The processes in steps S31 to S40 can be the same as in Figure 8.

[0174] S41: The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215, and the display control unit 13 displays the property display screen 220 including the first display area 214 and the second display area 215 (see Figure 11). For example, a list of items 221 placed on the property, three-dimensional image information 222 of the property, and a captured image (layout diagram) are displayed. Next, the user identifies an arbitrary item from the three-dimensional image information. Once the user identifies an item, they can request inspection information and a captured image of the item. The input receiving unit 12 of the terminal device 10 accepts the operation to identify an item. The item may be identified, for example, by the coordinates clicked by the user, or model identification information may be identified by these coordinates. The user also inputs comments (text information, audio) about the item to the terminal device 10, as described in Figures 6 and 7. Comments are sometimes referred to as input information. These comments may be tacit knowledge comments. The comments may also include caption comments that describe the item.

[0175] S42: The transmitting / receiving unit 11 of the terminal device 10 transmits information identifying the item and input information to the image management server 40.

[0176] The subsequent steps S43 to S46 can be the same as in Figure 8.

[0177] S47: The transmitting / receiving unit 41 of the image management server 40 receives inspection information and captured images from the terminal device 10. The storage / reading unit 49 identifies the item using model identification information and saves the latest inspection information and captured images in the image information management DB 4001. The judgment unit 43 obtains the caption comment identified by the model identification information from the caption model 4003 and determines the degree of relevance between the caption comment and the comments included in the input information received in step S42. The judgment unit 43 may determine the degree of relevance between the acquired caption comment and all comments included in the input information received in step S42, or it may divide the comments included in the input information received in step S42 into multiple parts and determine the degree of relevance between the acquired caption comment and each divided comment.

[0178] S48: The update unit 46 updates the caption model 4003 by associating the comments deemed highly relevant in step S47 with the model identification information as caption comments. The update unit 46 also updates the tacit knowledge model 4004 using the comments and inspection information deemed less relevant in step S47, along with the related three-dimensional image information of the item (identified in step S42) and captured image (received in S46), as training data. In other words, the correspondence between the three-dimensional image information and captured image of the item and the comments and inspection information is learned. Features are extracted from the three-dimensional image information and captured image of the item using several feature extraction models suitable for images, such as CNNs. Features represent, for example, what is pictured and where, or the content of the work being performed. Therefore, the tacit knowledge model 4004 can learn the correspondence between the features of the three-dimensional image information and captured image of the item and the comments and inspection information.

[0179] Note that comments and inspection information are not both necessary; at least one of them is sufficient to update the tacit knowledge model 4004.

[0180] Furthermore, in Figure 15, the image management server 40 acquires inspection information and captured images from the terminal device 10, but as shown in Figure 14, they may also be acquired from the property information management server 20.

[0181] <<Example of a screen during the learning phase>> The screen displayed by the terminal device 10 during the learning phase will be the same as in Figures 9 to 11, but in Figure 11, the user can input information. In addition, the information corresponding to Figure 12, which displays inspection information and captured images, will be displayed in the inference phase described below.

[0182] Figure 16 shows an example of a property display screen 220 displayed by the terminal device 10. The property display screen 220 in Figure 16 includes a first display area 214 and a second display area 215. The first display area 214 displays property information such as the size (floor area) 224 and a captured image 251. Similar to Figure 11, the captured image is a layout diagram.

[0183] The second display area 215 shows the three-dimensional image information 223 of the table, the input information 241 that corresponds to the three-dimensional image information 223 of the table, which states, "This table has an unstable center of gravity, so it is best not to place anything weighing more than 50 kg on it," and the captured image 252. Similar to Figure 11, the captured image 252 is a layout diagram. The image management server 40 can update the tacit knowledge model 4004 using this input information 241, inspection information, three-dimensional image information 223, and captured image 252. In addition, the area (floor area) 224 can be used as a caption comment as information about the property.

[0184] <Inference Phase (Generation of Text Information)> Next, with reference to Figure 17, the text information generation process using the tacit knowledge model 4004 will be explained. Figure 17 is a sequence diagram illustrating an example of the text information generation process. Note that the explanation of Figure 17 will mainly focus on the differences from Figure 8. The processes in steps S31 to S46 are the same as in Figure 8. However, in step S41, the user inputs a question about an item (see Figure 18).

[0185] S51: The transmitting / receiving unit 41 of the image management server 40 receives inspection information and captured images. The storage / reading unit 49 identifies the item using model identification information and saves the latest inspection information and captured images in the image information management DB 4001. The processing unit 47 requests the text information generation unit 45 to generate text information. The text information generation unit 45 obtains tacit knowledge comments corresponding to the three-dimensional image information and captured images of the item from the tacit knowledge model 4004. The tacit knowledge model 4004 can extract features of the three-dimensional image information and captured images of the item and identify at least one of the comments and inspection information corresponding to these features. The tacit knowledge model 4004 extracts at least one of the comments and inspection information as tacit knowledge comments. S52: Next, the text information generation unit 45 obtains text information created by the large-scale language model using the tacit knowledge comments, input information (question text), and check information. The large-scale language model 4005 can use these to generate more detailed text information. The text information generation unit 45 may also convert the audio information contained in the input information into text information, and the text information generated by the text information generation unit 45 may be either audio information or text information.

[0186] The text information generation unit 45 may generate text information without using any inspection information or question text. Alternatively, the text information generation unit 45 may generate fixed questions internally within the system and use these fixed questions. In this case, the question text is not visible to the user. Or, the text information generation unit 45 may generate fixed questions internally within the system, display these fixed questions on the display unit for the user to select, and then use the selected question.

[0187] While inspection information is not mandatory, using it to generate text information from the large-scale language model 4005 provides more detailed information about the item. For example, if the inspection information includes the extent of damage to the item, it is possible to generate text information that includes appropriate responses based on the extent of the damage.

[0188] S53: The processing unit 47 associates the three-dimensional image information of the item corresponding to the model identification information with the captured image and text information, and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to a second display area 215 that displays the three-dimensional image information of the item, the captured image, and text information corresponding to the three-dimensional image information of the item and the captured image. The screen generation unit 42 may also perform an update process to add only the text information to the screen corresponding to the second display area 215. The transmitting / receiving unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0189] S54: The display control unit 13 of the terminal device 10 displays an inspection image screen 230 which includes a first display area 214 and a second display area 215 which includes text information (see Figure 19). Alternatively, the conversion unit 15 converts the received text information into audio information, and the audio control unit 14 plays the converted text information to the speaker 109a. If the received text information is audio information, the text information is played to the speaker 109a, or the conversion unit 15 converts the received text information into character information and displays the converted text information on the display 106a.

[0190] In Figure 17, the image management server 40 acquires inspection information and captured images from the terminal device 10, but they may also be acquired from the property information management server 20, as shown in Figure 14.

[0191] <<Example of the inference phase screen>> The screen displayed by the terminal device 10 during the inference phase will be the same as in Figures 9 to 11, but in Figure 11, the user can input information (question text).

[0192] Figure 18 shows an example of a property display screen 220 in the inference phase. The property display screen 220 includes a first display area 214 and a second display area 215. Figure 18 has the same configuration as Figure 11, but the user has entered a question as input information (question text) 234. The second display area 215 displays the three-dimensional image information 223 of the table, the captured image 254, and the input information (question text) 234 corresponding to the three-dimensional image information 223 of the table. For example, the input information (question text) 234 in Figure 18 is "There is a scratch on the table, what should I do?" Along with such input information (question text) 234, the user presses the information display button 225 to request the generation of text information using the tacit knowledge model.

[0193] Figure 19 shows an example of text information displayed on the inspection image screen 230. The inspection image screen 230 includes a first display area 214 and a second display area 215. The second display area 215 displays three-dimensional image information 223 of the table, two captured images 254 and 258, and text information 235 corresponding to the three-dimensional image information 223 of the table. This text information 235 reads, "The scratch is less than 1 mm deep, so it will be repaired with paint. If it is 1 mm or deeper, it will be polished." The text information 235 was generated by the large-scale language model 4005 from the three-dimensional image information 223 of the item, inspection information, captured images 254, and input information (question text) 234. For example, if a scratch is detected on the item from the three-dimensional image information 223 or captured image 254, an implicit knowledge comment regarding the scratch on the item is extracted. This tacit knowledge comment, the question about the injury, and the inspection information identifying the current state of the injury are input into the large-scale language model 4005, which can then generate appropriate text information corresponding to the current injury.

[0194] The text information 235 is not the inspection information itself, but it includes at least one of the following: text information generated by an implicit knowledge model that has learned the inspection information, and text information generated by a large-scale language model 4005 based on the inspection information. Furthermore, the text information 235 can be considered the processing result of processing based on the inspection information and three-dimensional image information.

[0195] [Third Embodiment] In this embodiment, an image management server 40 that generates images from captured images and text information will be described.

[0196] Figure 20 is a functional configuration diagram illustrating an example of the functions of the image management server 40, the property information management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment. Note that the explanation of Figure 20 will mainly focus on the differences from Figure 3.

[0197] The image management server 40 in Figure 20 further includes an image generation unit 48, and an image generation model 4006 is constructed in the storage unit 4000 of the image management server 40. The other configurations may be the same as in Figure 3.

[0198] The image generation unit 48 is an example of an image generation means, and is implemented by instructions from the CPU 401 shown in Figure 2. It inputs text data or text data into the image generation model 4006 to generate image information.

[0199] Image generation model 4006 is a machine learning model (generative AI) that generates images from text data, or from text data and images. Image generation model 4006 is trained using training data that includes, for example, text data and images. This training data includes, for example, training text data, or text data and images as input, and ground truth images as output. For example, the image generation model 4006 may be trained so that the images it generates when inputting text data, or text data and images from the training data, approach the ground truth images contained in the training data.

[0200] <Learning Phase> The processing in the learning phase may be the same as in Figure 15. In step S48, the update unit 46 updates the tacit knowledge model 4004 to learn a correspondence between inputting comments and inspection information that were determined to have a low degree of relevance in step S47, and outputting three-dimensional image information or captured images of the item. Alternatively, the update unit 46 updates the tacit knowledge model 4004 to learn a correspondence between inputting comments, inspection information, and three-dimensional image information (or captured images) of the item, and outputting captured images (or three-dimensional image information).

[0201] <Inference Phase (Generation of Text Information)> Figure 21 is an example sequence diagram showing the process of generating text and image information. Note that the explanation of Figure 21 mainly focuses on the differences from Figure 17. In Figure 21, step S52-1 is added.

[0202] S52-1: The image generation unit 48 inputs the captured image and the text information created by the large-scale language model to the image generation model 4006 to generate image information. The image generation unit 48 may also obtain the image information created by the image generation model 4006 using the text information created by the large-scale language model without using the captured image.

[0203] The memory / reading unit 49 associates the inspection information stored in the image information management DB 4001 in step S46 with the captured images, and saves (or overwrites) the text information created by the large-scale language model and the image information created by the image generation model 4006 to the image information management DB 4001.

[0204] S53: The processing unit 47 associates the three-dimensional image information of the item corresponding to the model identification information with the generated image information and text information, and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the three-dimensional image information of the item, the generated image information, and text information corresponding to the three-dimensional image information of the item and the captured image. The transmitting / receiving unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmitting / receiving unit 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0205] <Example of the inference phase screen> Figure 22 shows the generated image information displayed on the inspection image screen 260. The explanation of Figure 22 mainly focuses on the differences from Figure 19.

[0206] The inspection image screen 260 in Figure 22 displays generated images 261 and 262. Generated images 261 and 262 are not the captured images 254 and 258 described in Figure 19, but rather generated images produced by the image generation model 4006 based on the captured images 254 and 258 and the text information 235. Therefore, generated images 261 and 262 have markers 263 and 264 that indicate the location of the damage. Note that one of the generated images 261 or 262 may be the captured image 254 or 258, or it may be possible to switch between them via user operation.

[0207] <Effects of generating text information using inspection data> The effects of generating text information using inspection information, as in this embodiment, will be explained.

[0208] 1. Comparative Example 1 (Using a General Large-Scale Language Model) Question: The user asks, "How should I repair the cracks?" Tacit comment: Tape or filler can be used.

[0209] 2. Comparative Example 2 (Case of learning from three-dimensional image information) • Learning Phase Input image: Three-dimensional image information Comment: Use tape for wide cracks and filler for narrow cracks. • Inference Phase Input image: Three-dimensional image information Question: "How should I repair the cracks?" Implicit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter.

[0210] 3. This embodiment (three-dimensional image information and captured image) • Learning Phase Input image: Three-dimensional image information and captured image Inspection information: Cracks found in the tape at the corners. • Inference Phase Input image: Three-dimensional image information and captured image Question: "How should I repair the cracks?" Tacit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter. However, when applying tape to corners, there is a risk of cracking, so please apply with care. In other words, "When applying tape to corners, there is a risk of cracking, so please apply with care" is the effect of learning the inspection information.

[0211] 4. This embodiment (three-dimensional image information, captured image, input information) • Learning Phase Input image: Three-dimensional image information and captured image Input information: Use tape for wide cracks and filler for narrow cracks. Inspection information: Cracks found in the tape at the corners. • Inference Phase Input image: Three-dimensional image information and captured image Question: "How should I repair the cracks?" Tacit knowledge comment: There are wide cracks and narrow cracks, so we recommend using tape for the former and filler for the latter. However, please be careful when applying tape to corners as there is a risk of cracking. In other words, the effect of learning the inspection information is that "please be careful when applying tape to corners as there is a risk of cracking."

[0212] <Multimodal> Several examples of combinations of input information and tacit knowledge comments are described. While the model described above assumed a large-scale language model, this embodiment can utilize a multimodal model that takes multiple data formats (images, text, gestures, etc.) as input and outputs them in a predetermined data format. • When the input information is a string, and content other than text information is generated as an implicit knowledge comment. Enter text to generate an image Enter text to generate a video. Enter text to generate speech. Enter a string of text to generate a 3D model. • When input information includes both strings and non-strings, and text information is generated as an implicit knowledge comment. Enter an image and text to generate text information. Enter a 3D model and text to generate text information. Input speech and text to generate text information. • When input information includes both text and non-textual content, and non-textual content is generated as an implicit knowledge comment. Enter an image and text to generate an image. Enter a video and text to generate a video. Enter a 3D model and text to generate a 3D model. Input speech and text to generate speech. <Main effects> In this embodiment, the image management server updates the tacit knowledge model based on inspection information, captured images, and three-dimensional image information, so that the terminal device 10 can display tacit knowledge comments corresponding to the inspection information, captured images, and three-dimensional image information. Even if inspection information and captured images cannot be obtained when generating text information, the tacit knowledge model can output tacit knowledge comments that take the inspection information into account based on the obtained three-dimensional image information and captured images.

[0213] [Other application examples] The present invention is not limited to the embodiments specifically disclosed above, and various modifications and changes are possible without departing from the scope of the claims. It goes without saying that the image management server 40 described in this embodiment is just one example, and there are various system configurations depending on the application and purpose.

[0214] For example, in this embodiment, we described an example in which an tacit knowledge model for industries such as civil engineering and construction answers a question, but the tacit knowledge model can be used in any industry where tacit knowledge is effective, such as healthcare, dental care, and investment decisions.

[0215] Furthermore, in this embodiment, the large-scale language model 4005 generates text information based on tacit knowledge comments, but it is also possible to use tacit knowledge comments as text information without using the large-scale language model 4005.

[0216] Furthermore, the tacit knowledge model 4004 may also be trained to take three-dimensional image information and inspection information as inputs, and output tacit knowledge comments using the input information. In other words, the tacit knowledge model 4004 may take information in different forms, such as images and text, as inputs.

[0217] Furthermore, although this embodiment describes a client-server type information processing system 100, the functions of the image management server 40 may be installed as an application on the terminal device 10. In other words, users may use the functions of this embodiment in a standalone format.

[0218] Furthermore, the configuration examples shown in Figure 3 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the image management server 40. The present invention is not limited by the way the processing units are divided or their names. The processing of the image management server 40 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit includes even more processing.

[0219] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0220] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the image management server 40 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0221] Furthermore, the image management server 40 can combine the disclosed processing steps in various ways. Each element of the image management server 40 may be combined into a single device or divided into multiple devices. Also, each processing performed by the image management server 40 may be performed by the terminal device 10.

[0222] <Nature> [Aspect 1] An information processing system comprising: a first server for managing information about an article, including a two-dimensional image of the article; a second server for managing three-dimensional image information of the article; and a terminal device capable of communicating with the first server and the second server, The terminal device includes a display control unit that displays a display screen containing information about the article received from the first server and three-dimensional image information of the article received from the second server. The second server includes a processing unit that performs a process to associate the three-dimensional image information of the article with the information of the article or generated information generated based on the information of the article, based on the information of the article transmitted by the first server and the three-dimensional image information of the article. The display control unit of the terminal device displays the three-dimensional image information of the article and the information relating to the article or the generated information corresponding to the three-dimensional image information, which have been received from the second server, on the display screen. Information processing system. [Aspect 2] The display control unit of the terminal device includes a first display area for displaying information about the article received from the first server, The display screen includes three-dimensional image information of the article received from the second server, and a second display area that displays information about the article corresponding to the three-dimensional image information or the generated information. The information processing system described in Embodiment 1. [Aspect 3] The second server stores the information about the article transmitted by the first server in a storage unit, associating it with the three-dimensional image information of the article. The information processing system according to embodiment 1 or 2. [Aspect 4] The processing unit requests information about the article from the first server and obtains information about the article from the first server as a response to the request. The information processing system described in Embodiment 1. [Aspect 5] The information relating to the said article includes inspection information of the said article and a two-dimensional image of the said article. An information processing system as described in any one of the four descriptions (1 to 4). [Aspect 6] The second server has a model that has learned the correspondence between the three-dimensional image information of the article, the two-dimensional image of the article, and the inspection information of the article. The processing unit acquires text information generated by the model as generated information, based on the three-dimensional image information and the two-dimensional image of the article selected by the terminal device. The information processing system described in aspect 5. [Aspect 7] The second server has a model that has learned the correspondence between the three-dimensional image information and two-dimensional image of the article and the inspection information and input information received from the terminal device. The processing unit acquires text information generated by the model as generated information, based on the three-dimensional image information and the two-dimensional image of the article selected by the terminal device. The information processing system described in aspect 5. [Aspect 8] The second server has an update unit that updates the model by learning the correspondence between the three-dimensional image information of the article, the two-dimensional image of the article, and the inspection information of the article. The information processing system described in Embodiment 6. [Aspect 9] The second server has an update unit that updates the model by learning the correspondence between the three-dimensional image information of the article and the two-dimensional image of the article, the inspection information of the article, and the input information received from the terminal device. The information processing system described in Embodiment 7. [Aspect 10] The three-dimensional image information of the article displayed by the display control unit is a two-dimensional projection of the three-dimensional model shape of the article, and the image of the article can be displayed by changing the viewpoint. An information processing system as described in any one of the descriptions 1 to 9. [Aspect 11] The aforementioned terminal device is The display control unit has an input receiving unit that, while displaying the three-dimensional image information of the article and information relating to the article corresponding to the three-dimensional image information or the generated information on the display screen, accepts the selection of an article other than the article on which the three-dimensional image information is displayed. The processing unit performs a process to associate the three-dimensional image information of the other article with the information of the other article or generated information generated based on the information of the other article, based on the information of the other article transmitted by the first server and the three-dimensional image information of the other article. The display control unit displays the three-dimensional image information of the other article received from the second server, and the information relating to the other article corresponding to the three-dimensional image information, or the generated information, on the display screen. An information processing system as described in any one of the descriptions 1 to 10. [Explanation of Symbols]

[0223] 10 Terminal devices 20 Property Information Management Server 40 Image Management Server 100 Information Processing Systems [Prior art documents] [Patent Documents]

[0224] [Patent Document 1] Patent No. 6954410

Claims

1. An information processing system comprising: a first server for managing information about an article, including a two-dimensional image of the article; a second server for managing three-dimensional image information of the article; and a terminal device capable of communicating with the first server and the second server, The terminal device includes a display control unit that displays a display screen containing information about the article received from the first server and three-dimensional image information of the article received from the second server. The second server includes a processing unit that performs a process to associate the three-dimensional image information of the article with the information of the article or generated information generated based on the information of the article, based on the information of the article transmitted by the first server and the three-dimensional image information of the article. The display control unit of the terminal device displays the three-dimensional image information of the article and the information relating to the article or the generated information corresponding to the three-dimensional image information, which have been received from the second server, on the display screen. Information processing system.

2. The display control unit of the terminal device includes a first display area for displaying information about the article received from the first server, The display screen includes three-dimensional image information of the article received from the second server, and a second display area that displays information about the article corresponding to the three-dimensional image information or the generated information. The information processing system according to claim 1.

3. The second server stores the information about the article transmitted by the first server in a storage unit, associating it with the three-dimensional image information of the article. The information processing system according to claim 1 or 2.

4. The processing unit requests information about the article from the first server and obtains information about the article from the first server as a response to the request. The information processing system according to claim 1.

5. The information relating to the said article includes inspection information of the said article and a two-dimensional image of the said article. The information processing system according to claim 1.

6. The second server has a model that has learned the correspondence between the three-dimensional image information of the article, the two-dimensional image of the article, and the inspection information of the article. The processing unit acquires text information generated by the model as generated information, based on the three-dimensional image information and the two-dimensional image of the article selected by the terminal device. The information processing system according to claim 5.

7. The second server has a model that has learned the correspondence between the three-dimensional image information and two-dimensional image of the article and the inspection information of the article and input information received from the terminal device. The processing unit acquires text information generated by the model as generated information, based on the three-dimensional image information and the two-dimensional image of the article selected by the terminal device. The information processing system according to claim 5.

8. The second server has an update unit that updates the model by learning the correspondence between the three-dimensional image information of the article, the two-dimensional image of the article, and the inspection information of the article. The information processing system according to claim 6.

9. The second server has an update unit that updates the model by learning the correspondence between the three-dimensional image information of the article and the two-dimensional image of the article, the inspection information of the article, and the input information received from the terminal device. The information processing system according to claim 7.

10. The three-dimensional image information of the article displayed by the display control unit is a two-dimensional projection of the three-dimensional model shape of the article, and the image of the article can be displayed by changing the viewpoint. The information processing system according to claim 1.

11. The aforementioned terminal device is The display control unit has an input receiving unit that, while displaying the three-dimensional image information of the article and information relating to the article corresponding to the three-dimensional image information or the generated information on the display screen, accepts the selection of an article other than the article on which the three-dimensional image information is displayed. The processing unit performs a process to associate the three-dimensional image information of the other article with the information of the other article or generated information generated based on the information of the other article, based on the information of the other article transmitted by the first server and the three-dimensional image information of the other article. The display control unit displays the three-dimensional image information of the other article received from the second server, and the information relating to the other article corresponding to the three-dimensional image information, or the generated information, on the display screen. The information processing system according to claim 1.

12. The second server in a system comprising: a first server for managing information about an article, including a two-dimensional image of the article; a second server for managing three-dimensional image information of the article; and a terminal device capable of communicating with the first and second servers and displaying a display screen including information about the article received from the first server and image information of the article received from the second server, A processing unit that performs a process to associate the three-dimensional image information of the article with the information of the article or generated information generated based on the information of the article, based on the information of the article transmitted by the first server and the three-dimensional image information of the article. A transmission unit transmits to the terminal device screen information for displaying three-dimensional image information of the article and information relating to the article corresponding to the three-dimensional image information or generated information on the display screen, A second server having [a certain feature].

13. The first server in a system comprising a first server for managing information about an article, including a two-dimensional image of the article; a second server for managing three-dimensional image information of the article; and a terminal device that can communicate with the first and second servers and displays a display screen including information about the article received from the first server and three-dimensional image information of the article received from the second server, The system includes a transmission unit that transmits information about the item to the terminal device in order to display it on the aforementioned display screen. The transmitting unit further transmits information about the article to the terminal device in response to a request from the terminal device that has received a request from the second server, or transmits information about the article to the second server in response to a request from the second server, in order to display the three-dimensional image information of the article and the information about the article corresponding to the three-dimensional image information of the article or generated information generated based on the information about the article on the display screen. The first server.

14. An information processing method performed by the second server in a system comprising: a first server for managing information about an article, including a two-dimensional image of the article; a second server for managing three-dimensional image information of the article; and a terminal device capable of communicating with the first and second servers and displaying a display screen including information about the article received from the first server and image information of the article received from the second server, wherein the second server performs information processing, A process that performs a process to associate the three-dimensional image information of the article with the information of the article or generated information generated based on the information of the article, based on the information of the article transmitted by the first server and the three-dimensional image information of the article. A process of transmitting to the terminal device screen information for displaying on the display screen the three-dimensional image information of the article and the information relating to the article corresponding to the three-dimensional image information or the generated information, An information processing method that performs this task.

15. In a system including a first server that manages information about an article, including a two-dimensional image of the article, a second server that manages three-dimensional image information of the article, and a terminal device that can communicate with the first and second servers and displays a display screen including information about the article received from the first server and image information of the article received from the second server, the second server A process that performs a process to associate the three-dimensional image information of the article with the information of the article or generated information generated based on the information of the article, based on the information of the article transmitted by the first server and the three-dimensional image information of the article. A process of transmitting to the terminal device screen information for displaying on the display screen the three-dimensional image information of the article and the information relating to the article corresponding to the three-dimensional image information or the generated information, A program that executes the command.

Images