Information processing system, first server, second server, information processing method, and program

Abstract

Processing is performed based on information managed by the first server and information managed by the second server without adding any processing function to the first server. [Solution] A system 100 includes a first server (property information management server) that manages information about items, a second server (image management server) that manages image information about items, and a terminal device that can communicate with the first and second servers, wherein the terminal device has a display control unit that displays a display screen including information about items received from the first server and image information about items received from the second server, and the second server has a processing unit 47 that associates the image information about items with information about items or generated information generated based on the information about items based on the information about items, based on the information about items transmitted by the first server and the image information about items. The display control unit of the terminal device displays the image information about items acquired from the second server and the information about items or generated information corresponding to the image information on the display screen.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a first server, a second server, an information processing method, and a program. [Background technology]

[0002] In some cases, a first server and a second server each manage related information. In such cases, a technique is known in which a terminal device displays the information managed by the first server and the information managed by the second server (see, for example, Patent Document 1).

[0003] Patent Document 1 discloses a system that displays, on a communication terminal, property-related information transmitted from a link information management system and a spherical image of the property transmitted from an image management system. Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention provides a technique for performing processing based on information managed by a first server and information managed by a second server without adding processing functions to the first server. [Means for solving the problem]

[0005] In view of the above problems, the present invention provides a system including a first server that manages information about an item, a second server that manages image information about the item, and a terminal device that can communicate with the first server and the second server, The terminal device a display control unit that displays a display screen including information about the item received from the first server and image information about the item received from the second server, the second server includes a processing unit that executes a process of associating the image information of the item with the information about the item or generation information generated based on the information about the item, based on the information about the item and image information about the item transmitted by the first server; The display control unit of the terminal device displays, on the display screen, image information of the item received from the second server, and information about the item corresponding to the image information or the generated information. [Effects of the Invention]

[0006] Processing can be performed based on information managed by the first server and information managed by the second server without adding any processing function to the first server. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an overall configuration of an example of an information processing system. [Figure 2] FIG. 2 is a hardware configuration diagram of an example of an image management server, a property information management server, or a terminal device. [Figure 3] 1 is a functional configuration diagram illustrating an example of functions of an image management server, a property information management server, and a terminal device in an information processing system. [Figure 4] FIG. 2 is a conceptual diagram illustrating an example of an image information management table according to the present embodiment. [Figure 5] FIG. 2 is a conceptual diagram illustrating an example of a property management information management table according to the present embodiment. [Figure 6] 10A and 10B are explanatory diagrams displayed on the terminal device in the text information generation process and the model update process. [Figure 7] 10A and 10B are explanatory diagrams displayed on the terminal device in the text information generation process and the model update process. [Figure 8] FIG. 10 is a sequence diagram showing an example of a process for generating screen information in which inspection information and image information are arranged as a process based on both the inspection information and the image information. [Figure 9] FIG. 10 illustrates an example property management screen. [Figure 10] FIG. 10 illustrates an example property management screen. [Figure 11] FIG. 10 is a diagram showing an example of a property display screen. [Figure 12] FIG. 10 is a diagram showing an example of an inspection image screen. [Figure 13] 10 is a diagram illustrating an example of switching between image information and inspection information of an article displayed in the second display area. FIG. [Figure 14] FIG. 10 is a sequence diagram illustrating an example of a process for generating screen information based on inspection information and image information. [Figure 15] FIG. 10 is a sequence diagram illustrating an example of a model update process. [Figure 16] FIG. 10 is a diagram showing an example of a property display screen displayed by the terminal device. [Figure 17] FIG. 10 is a sequence diagram illustrating an example of a text information generation process. [Figure 18] FIG. 10 is a diagram showing an example of a property display screen in the inference phase. [Figure 19] FIG. 10 is a diagram showing an example of text information displayed on an inspection image screen. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An information processing system and an information processing method performed by the information processing system will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings.

[0009] <Additional information on tacit knowledge> In industries such as civil engineering and construction, BIM / CIM is being promoted in order to address issues such as the declining birthrate and aging population, and to improve labor productivity.

[0010] BIM is an abbreviation for Building Information Modeling, and is a solution that utilizes information in all processes from building design and construction to maintenance and management by using a building database that adds attribute data such as cost, finish, and management information to a three-dimensional digital model of a building (hereinafter referred to as a 3D model) created on a computer.

[0011] CIM is an abbreviation for Construction Information Modeling, and is a solution for the civil engineering field (general infrastructure such as roads, electricity, gas, and water) that was proposed following the example of BIM, which was being promoted in the construction field. Like BIM, CIM is being developed to improve the efficiency and sophistication of the entire construction production system by sharing information between stakeholders, mainly through 3D models.

[0012] The key to promoting BIM and CIM is how to utilize the BIM and CIM that have been created.

[0013] Specifically, 3D models restored using BIM and CIM can be used not only for design and construction purposes, but also for other tasks such as maintenance and site inspections. In other words, in addition to using them as blueprints, other uses such as keeping records in the 3D model and sharing it with others can be considered.

[0014] Furthermore, because the work performed on the 3D model can be recorded as a log, if tacit knowledge can be extracted from this, it can be effectively used to transfer skills from experienced workers to younger workers, which is expected to lead to front-loading of work and human resource development.

[0015] Here, when we focus on the transfer of tacit knowledge, as mentioned above, the challenge is how to transfer tacit knowledge between different tasks and between users with different levels of expertise, not only for 3D models but also for 2D data (such as spherical images or flat images).

[0016] Specifically, because tacit knowledge is qualitative and difficult to quantify, even if a tacit knowledge model is generated from tacit knowledge, it is difficult to ensure users' trust in the tacit knowledge model, making it difficult to promote the use of the tacit knowledge model. For example, if the tacit knowledge model's field of expertise is different from that of the user, no matter how excellent the tacit knowledge model is, it will have no value to the user. Similarly, if the knowledge level of the tacit knowledge model is lower than that of the user, it will have no value to the user.

[0017] However, it is also true that tacit knowledge models can provide users with new perspectives and insights, and by using tacit knowledge models, even inexperienced users have the potential to acquire know-how and skills and apply them in their work.

[0018] Furthermore, there may be a case where it is desired to add a function to display images such as 3D models corresponding to the property management information to a system including a first server that stores property management information and a terminal device.

[0019] One way to achieve this would be for the first server to acquire all of the property management information and image information, but this would require additional functionality to the first server, which would increase costs.

[0020] Therefore, in this embodiment, the second server executes processing based on the property management information managed by the first server and the image information of the property managed by the second server, so that the user can display the image information and the property management information acquired from the first server on a single screen in association with each other on the terminal device. Furthermore, the second server not only displays two pieces of information on the terminal device, but can also display tacit knowledge about the property (such as text information) generated based on the property management information and the image information in association with the image information. This allows the terminal device to display the image information and the property management information acquired from the first server in association with each other on a single screen, or to display tacit knowledge about the property in association with the image information, without adding significant functionality to the first server.

[0021] <Terminology> A user is someone who uses the text information generated by the tacit knowledge model (it may also output content other than text, such as images). A data provider is someone who provides data (audio information, character information, operation information, images, 3D data, etc.) used by the tacit knowledge model for learning.

[0022] Tacit knowledge is knowledge based on personal experience and intuition. A tacit knowledge model is a model that learns tacit knowledge and outputs answers to questions based on the learned tacit knowledge. A model is a mechanism or artificial intelligence (AI) that learns the correspondence between input data and output data and outputs output data in response to the input data. Note that the output data can be generated regardless of whether or not training data is used.

[0023] A property is a space where items can be placed, such as a facility or a room within a facility. Items are objects placed in the property. The types of items placed vary depending on the function of the facility.

[0024] For example, properties include real estate, factories, construction sites, research facilities, medical facilities, farmland, warehouses, equipment that requires maintenance, etc. Goods include furniture, materials, equipment, heavy machinery, tools, instruments, ingredients, cultures, food, etc.

[0025] Information about an item includes the name of the item, the name of the property where the item is located, etc., but is preferably information that can become tacit knowledge. For example, information about an item can be information that is known only to a certain user. In this embodiment, information about an item is described using the term inspection information. In addition, information about an item may include an instruction manual for the item, a daily report of work performed on the item, an estimate for the item, a drawing of the item, etc. Furthermore, inspection may include investigation, testing, verification, examination, checking, etc.

[0026] The image information of the item is an image of the item displayed on a display. The image information of the item may be a two-dimensional display or a three-dimensional display. The image information of the item may also be a spherical image.

[0027] The display screen (including information about the item and image information about the item) is, for example, a screen on which inspection information and an image of the item are displayed on one screen. In this embodiment, the screens shown in Figs. 12 and 19 are exemplified.

[0028] [First embodiment] <System configuration example> 1 is a diagram showing the overall configuration of an example of an information processing system 100. The information processing system 100 of this embodiment includes a terminal device 10, which is an example of an input / output device, an image management server 40, and a property information management server 20. However, the terminal device 10 may be connected to the image management server 40 or the property information management server 20 as needed, and may not be included in the information processing system 100.

[0029] The image management server 40 (an example of a second server) is one or more information processing devices that can communicate with the terminal device 10 via the communication network N. The image management server 40 manages image information of properties and has a tacit knowledge model and a large-scale language model. The image management server 40 uses these to return text information containing tacit knowledge to the user. The image management server 40 may be a web server that returns processing results to the terminal device 10 in response to a request from the terminal device 10. A server is a computer or software that performs the function of providing information and processing results in response to a request from a client.

[0030] The image management server 40 may be compatible with cloud computing. Cloud computing refers to a usage pattern in which resources on a network are used without regard to specific hardware resources. Cloud computing can take the form of SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service), but any of these forms is acceptable. Therefore, the image management server 40 does not need to be housed in a single housing or provided as a single integrated device. The functions of the image management server 40 may be distributed among multiple information processing devices, or multiple information processing devices may each have all of the functions and the information processing device that performs processing may be switched by load balancing or the like. The image management server 40 may also be a server located in an on-premises environment.

[0031] In addition, instead of the image management server 40 having the tacit knowledge model and the large-scale language model, the image management server 40 may call an API (Application Programming Interface) published by an external system and use at least one of the tacit knowledge model or the large-scale language model.

[0032] The property information management server 20 (an example of a first server) is one or more information processing devices that can communicate with the terminal device 10 via the communication network N. The property information management server 20 manages property management information. The property management information is, for example, a character string such as text, and the property information management server 20 does not have image information, or if it does, it is merely a photograph or the like that is different from the images managed by the image management server 40. The property information management server 20 is a server that allows users to update the property management information as appropriate and manage the progress of construction work on the property and the placement of items, etc.

[0033] The property information management server 20 may be a web server that returns processing results to the terminal device 10 in response to a request from the terminal device 10. The property information management server 20 can communicate with the image management server 40 via a communication network N. The property information management server 20 may also be compatible with either cloud computing or on-premise.

[0034] It is preferable that the image management server 40 and the property information management server 20 are linked to the extent that single sign-on is possible. The image management server 40 can communicate with the property information management server 20 via an API made public by the property information management server 20. Alternatively, the image management server 40 and the property information management server 20 may be in business partnership.

[0035] The terminal device 10 is a general-purpose information processing terminal used by a user of the information processing system 100. A web browser and a native application dedicated to the image management server 40 or the property information management server 20 run on the terminal device 10. When the terminal device 10 runs a web browser, the terminal device 10 and the image management server 40 or the property information management server 20 run a web application. A web application is an application that runs through cooperation between a program written in a programming language (e.g., JavaScript (registered trademark)) that runs on the web browser and a program on the web server (image management server 40) side. When the web application is executed, the processing of this embodiment may be performed by the image management server 40 or the property information management server 20, or may be performed by the terminal device 10 that has received the web application.

[0036] An application that cannot be executed unless it is installed on the terminal device 10 is called a native application. In this embodiment, too, the application executed on the terminal device 10 may be a web application or a native application. In this case, too, the processing of this embodiment may be performed by the image management server 40 or by the terminal device 10 that executes the native application.

[0037] The terminal device 10 is, for example, a PC (Personal Computer), a smartphone, a PDA (Personal Digital Assistant), a tablet terminal, etc. Alternatively, the terminal device 10 may be any device on which a web browser or a native application runs. The terminal device 10 may also be an electronic whiteboard, a television set, a glasses device, or a wearable device. Furthermore, there may be multiple terminal devices 10.

[0038] The terminal device 10 can communicate with the image management server 40 and the property information management server 20 via a communication network N. The communication network N is constructed using the Internet, a LAN (Local Area Network), provider services, etc. The communication network N may include not only wired communication but also mobile communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution), and networks using wireless LAN. The terminal device 10 can also communicate using short-range communication technologies such as Bluetooth (registered trademark) and NFC (registered trademark, Near Field Communication).

[0039] 1, the image management server 40, the property information management server 20, and the terminal device 10 communicate with each other via a communication network N. However, a user may operate the image management server 40 or the property information management server 20 directly from a console.

[0040] <Hardware configuration example> 2 is a diagram showing an example of the hardware configuration of the image management server 40, the property information management server 20, or the terminal device 10 according to this embodiment. Each hardware component of the image management server 40 or the property information management server 20 is indicated by a reference number in the 400s. Each hardware component of the terminal device 10 is indicated by a reference number in the 100s.

[0041] The hardware configuration of the terminal device 10 will be described below, but the hardware configuration of the image management server 40 or the property information management server 20 is similar, so the description will be omitted.

[0042] The terminal device 10 is constructed by a computer and, as shown in FIG. 2, includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a HD (Hard Disk) 104, a HDD (Hard Disk Drive) controller 105, a display I / F 106, and a communication I / F 107.

[0043] Of these, the CPU 101 controls the overall operation of the terminal device 10. The ROM 102 stores programs such as an IPL (Initial Program Loader) used to drive the CPU 101. The RAM 103 is used as a work area for the CPU 101.

[0044] The HD 104 stores various data such as programs, etc. The HDD controller 105 controls reading and writing of various data from and to the HD 10 under the control of the CPU 101.

[0045] The display I / F 106 is a circuit that displays images on a display 106a. The display 106a is a type of display unit, such as a liquid crystal display or organic electroluminescence (EL) display, that displays various information such as a cursor, menu, window, text, or image. The communication I / F 107 is an interface used for communication with other devices.

[0046] When the terminal device 10 is a glasses device, the terminal device 10 may use, instead of the display I / F 106, a circuit for displaying an image on a lens or the like as a transmissive / reflective member.

[0047] The communication I / F 107 is, for example, a network interface card (NIC) that supports transmission control protocol (TCP) / internet protocol (IP).

[0048] The terminal device 10 also includes a sensor I / F 108 , a sound input / output I / F 109 , an input I / F 110 , a media I / F 111 , and a DVD-RW (Digital Versatile Disk Rewritable) drive 112 .

[0049] The sensor I / F 108 is an interface that receives detection information from various sensors. The sound input / output I / F 109 is a circuit that processes input and output of sound signals between the speaker 109a and the microphone 109b under the control of the CPU 101. The input I / F 110 is an interface that connects a predetermined input means to the terminal device 10.

[0050] The keyboard 110a is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The mouse 110b is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, and performing operations on the display screen, etc.

[0051] The media I / F 111 controls reading and writing (storing) of data from and to a recording medium 111a such as a flash memory. The DVD-RW drive 112 controls reading and writing of various data from and to a DVD-RW 112a, which is an example of a removable recording medium. The DVD-RW may be a DVD-R or the like. The DVD-RW drive 112 may also be a Blu-ray drive that controls reading and writing of various data from and to a Blu-ray Disc (registered trademark).

[0052] The terminal device 10 also includes a bus line 113. The bus line 113 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 101.

[0053] The recording media, such as HDs and CD-ROMs, on which the above programs are stored can be provided domestically or internationally as program products. The terminal device 10, for example, realizes the information processing method according to the present invention by executing the program according to the present invention.

[0054] <About the function> FIG. 3 is a functional configuration diagram illustrating an example of functions possessed by the image management server 40, the property information management server 20, and the terminal device 10 in the information processing system 100 according to this embodiment.

[0055] <<Terminal Device>> As shown in Fig. 3, the terminal device 10 has a transmitting / receiving unit 11, an input receiving unit 12, a display control unit 13, an audio control unit 14, a conversion unit 15, and a storage / readout unit 19. Each of these units is a function or a means for performing the function, which is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 101 in accordance with a program loaded from the HD 104 onto the RAM 103. The terminal device 10 also has a storage unit 1000 constructed by at least one of the RAM 103 and the HD 104 shown in Fig. 2.

[0056] The transmitter / receiver unit 11 is an example of a transmitter or receiver unit, and is realized by instructions from the CPU 101 shown in Figure 2 and the communication I / F 107, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0057] The input reception unit 12 is an example of an input reception means, and is mainly realized by commands from the CPU 101 shown in FIG. 2, as well as the input I / F 110 and the sound input / output I / F 109, and receives various inputs from the user using the microphone 109b, keyboard 110a and mouse 110b.

[0058] The display control unit 13 is an example of a display control means and an output means, and is realized by commands from the CPU 101 shown in Fig. 2 and the display I / F 106, and causes the display 106a, which is an example of a display unit, to display various images and screens. When the terminal device 10 is a glasses device, the display control unit 13 causes a transmissive reflective member such as a lens to display a virtual image instead of the display I / F 106.

[0059] The audio control unit 14 is an example of an audio control means and output means, and is realized by instructions from the CPU 101 shown in Figure 2 and the audio input / output I / F 109, and causes the speaker 109a, which is an example of an audio playback unit, to play sound.

[0060] The conversion unit 15 is an example of a processing means, and is realized by an instruction from the CPU 101 shown in FIG. 2, and performs a process of converting character information into voice information and a process of converting voice information into character information.

[0061] The storage / reading unit 19 is an example of a storage control means, and is executed by commands from the CPU 101 shown in FIG. 2, as well as the HD 104, the media I / F 111, and the DVD-RW drive 112, to store various data in the storage unit 1000, the recording medium 111a, and the DVD-RW 112a, and to read various data from the storage unit 1000, the recording medium 111a, and the DVD-RW 112a.

[0062] <Functional configuration of the image management server> The image management server 40 has a transmission / reception unit 41, a screen generation unit 42, a determination unit 43, an identification unit 44, a text information generation unit 45, an update unit 46, a processing unit 47, and a storage / readout unit 49. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 401 in accordance with a program loaded from the HD 404 onto the RAM 403. The image management server 40 also has a memory unit 4000 constructed by the HD 404 shown in Fig. 2. The memory unit 4000 is an example of a memory means.

[0063] In FIG. 3, all functions are installed in one image management server 40, but the image management server 40 may be configured so that each function is distributed among a plurality of computers.

[0064] The transmitter / receiver unit 41 is an example of a transmitter or receiver unit, and is realized by instructions from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0065] The screen generation unit 42 is an example of a screen generation means, and is realized by commands from the CPU 401 shown in Fig. 2, to generate various screens. When the terminal device 10 executes a Web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), and the like. Therefore, the screen information may also be referred to as a Web application. When the terminal device 10 executes a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like.

[0066] The determining unit 43 is an example of a determining means, and is realized by an instruction from the CPU 401 shown in FIG. 2, and performs various determinations described below.

[0067] The identification unit 44 is an example of an identification means, and is realized by an instruction from the CPU 401 shown in FIG. 2, and identifies the target image.

[0068] The text information generation unit 45 is an example of a text information generation means, and is realized by an instruction from the CPU 401 shown in FIG. 2, and acquires tacit knowledge comments from the tacit knowledge model and generates text information based on the large-scale language model 4005.

[0069] The update unit 46 is an example of an update means, and is realized by an instruction from the CPU 401 shown in FIG. 2, and updates the tacit knowledge model, which will be described later.

[0070] The processing unit 47 performs processing to associate image information with text information, which is an example of inspection information or generated information generated based on inspection information, based on the inspection information and image information, in accordance with processing requested by the user. Processing based on the inspection information and image information includes processing to display both on one screen, or processing to acquire tacit knowledge comments, which are an example of text information, from the tacit knowledge model 4004 using the inspection information and image information. Note that the processing unit 47 requests processing from the screen generation unit 42, the text information generation unit 45, or the like, in accordance with the content of the processing. Note that the processing unit 47 may perform processing to associate image information with generated image information, which is another example of generated information generated based on the inspection information, in accordance with the purpose of the generated information generated based on the inspection information.

[0071] The storage / readout unit 49 is an example of storage control means, and is executed by commands from the CPU 401 shown in Fig. 2, as well as the HD 404, media I / F 411, and DVD-RW drive 412, to store various data in the storage unit 4000, recording medium 411a, and DVD-RW 412a, and to read various data from the storage unit 4000, recording medium 411a, and DVD-RW 412a. The storage unit 4000, recording medium 411a, and DVD-RW 412a are examples of storage means.

[0072] In the storage unit 4000, an image information management DB 4001, a model shape management DB 4002, a caption model 4003, a tacit knowledge model 4004, and a large-scale language model 4005 are constructed.

[0073] The image information management DB 4001 manages image information related to properties. This image information is information that visually represents an item (also called a model) placed on a property. The model shape management DB 4002 manages three-dimensional model shape information of the item placed on a property. The image management server 40 can generate image information related to properties using the three-dimensional model shape information. The three-dimensional model shape information is information that depicts an item in three dimensions, such as a three-dimensional model or three-dimensional point cloud of the item. The three-dimensional model shape information may be configured using polygons, CAD, or the like. Furthermore, the image information management DB 4001 or the model shape management DB 4002 may store a wide-field image, such as a spherical image of the property.

[0074] The caption model 4003 is generated by executing a learning process using a combination of an image and a caption comment as learning data, and is a model that causes a computer to function to output a caption comment based on the image. A caption comment is explicit knowledge and is used as a term corresponding to tacit knowledge. Here, a caption comment is text data, and is, for example, a comment expressed in audio or text that explains an image. A caption comment related to an object or item is associated with identification information of the object or item.

[0075] The tacit knowledge model 4004 is a model that is generated by executing a learning process using as learning data the correspondence between image information and comments on this image information (comments other than caption comments), and causes a computer to function to output tacit knowledge comments based on images. The tacit knowledge model 4004 may also learn the correspondence between information other than comments and image information, as described below. Correspondence between image information and inspection information Correspondence between image information, comments, and inspection information Here, the tacit knowledge comment is text data, and is a comment expressed by voice or text, excluding caption comments, that is, a comment relating to content not expressed in an image.

[0076] The large-scale language model 4005 is a computer language model that is generated by executing a learning process using a huge amount of unlabeled text as training data and is composed of an artificial neural network with many parameters. The large-scale language model 4005 can capture much of the syntax and meaning of human language by being sufficiently trained using a method for learning context, such as next sentence prediction, which understands the context by determining whether sentence 1 and sentence 2 are consecutive, or a masked language model, which understands the context by masking words in a sentence and predicting the masked words from the words before and after them.

[0077] <<Image information management table>> 4 is a conceptual diagram showing an example of an image information management table according to this embodiment. An image information management DB 4001 configured with an image information management table such as that shown in FIG. 4 is created in the storage unit 4000.

[0078] In this image information management table, model identification information, location information, item information, inspection information, and image information are managed in association with property identification information. A property can be any space in which an item can be placed, such as a facility or a room within a facility. The types of items that can be placed vary depending on the function of the facility.

[0079] · Property identification information is identification information that uniquely identifies a property.

[0080] The model identification information is identification information that identifies an item placed in a property. The model identification information associates the property management information with the three-dimensional model shape in the model shape management DB 4002. This model identification information is the same as the model identification information in the property information management server 20, but may also be associated with the model identification information in the property information management server 20 in a conversion table.

[0081] Item information is the item name of the item placed in the property. In Figure 4, item names are displayed assuming a construction site or building site, but item names vary depending on the property. Also, in Figure 4, the property and item are associated, but the item may be registered independently without any relation to the property.

[0082] Location information is information that indicates the location of an item in a virtual space using three-dimensional coordinates (X, Y, Z). X, Y, Z may be the center of gravity of the item, or the coordinates of the vertices of a circumscribing cube.

[0083] Inspection information is information relating to inspections performed by users on items in a property. Inspection information varies depending on the item and the purpose of the inspection, but may include, for example, whether there are any abnormalities with the item, its dimensions, the date and time of inspection, and the inspector. Inspection information is obtained from the property information management server 20. The property information management server 20 holds master inspection information, and the image management server 40 obtains and stores the inspection information from the property information management server 20. This is because inspections of items are performed on the property information management server 20. For this reason, inspection information in the image information management table may only exist for some items, or even if inspection information exists, it may not be the most recent inspection information.

[0084] The inspection information may also be photographic information of the inspected item. In this case, the update unit 46 analyzes the photographic information, extracts features of the photographed item, the state of any damage, etc., and converts the extracted features into inspection information represented by a character string.

[0085] In addition to inspection information, instruction manuals, daily reports, estimates, drawings, etc. may also be registered in the image information management table.

[0086] Image information is information that visually represents an article, such as three-dimensional model shape information of the article, a spherical image, etc. In Fig. 4, the image information is arranged in the image information management table, but the image information may be stored in the model shape management DB 4002.

[0087] <Functional configuration of property information management server> Returning to Fig. 3, the property information management server 20 has a transmission / reception unit 21, a screen generation unit 22, and a storage / readout unit 29. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 401 in accordance with a program loaded from the HD 404 onto the RAM 403. The property information management server 20 also has a memory unit 2000 constructed by the HD 404 shown in Fig. 2. The memory unit 2000 is an example of a memory means.

[0088] In FIG. 3, all functions are installed in one property information management server 20, but the property information management server 20 may be configured such that each function is distributed among a plurality of computers.

[0089] The transmitter / receiver unit 21 is an example of a transmitter or receiver unit, and is realized by instructions from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network N.

[0090] The screen generation unit 22 is an example of a screen generation means, and is realized by commands from the CPU 401 shown in Fig. 2, to generate various screens. When the terminal device 10 executes a Web application, the screen information is created using HTML, XML, CSS, JavaScript (registered trademark), and the like. Therefore, the screen information may also be referred to as a Web application. When the terminal device 10 executes a client application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like.

[0091] The storage / readout unit 29 is an example of storage control means, and is executed by commands from the CPU 401 shown in Fig. 2, as well as the HD 404, media I / F 411, and DVD-RW drive 412, to store various data in the storage unit 2000, recording medium 411a, and DVD-RW 412a, and to read various data from the storage unit 2000, recording medium 411a, and DVD-RW 412a. The storage unit 2000, recording medium 411a, and DVD-RW 412a are examples of storage means.

[0092] <<Property Management Information Management Table>> 5 is a conceptual diagram showing an example of a property management information management table according to this embodiment. A property information management DB 2001 configured by the property management information management table shown in FIG. 5 is created in the storage unit 2000.

[0093] In this property management information management table, model identification information, location information, item information, and inspection information are managed in association with property identification information. These are sometimes referred to as property management information. The contents of these may be the same as those of the image information management table. That is, the property management information management table does not hold image information. However, inspection information is recorded for all items that have been inspected at least once. Furthermore, since the inspection information in the property information management server 20 is inspection information registered directly by the user, it serves as the master (copy source) of the inspection information in the image management server 40. Note that instead of the property information management DB2001 configured with the above property management information management table, a property information management DB2001 may be used that is configured with a property management information management table in which location information, item information, and inspection information are managed in association with model identification information without being associated with property identification information.

[0094] <An example of updating a model and generating text information> The method for updating the model and generating text information will be described with reference to Figures 6 and 7. In Figures 6 and 7, inspection information is not used to update the model and generate text information, but similar learning can be achieved by replacing or adding inspection information to the following dialogue Q1, etc.

[0095] 6A and 6B are explanatory diagrams of the display on the terminal device 10 in the text information generation process and the model update process. Fig. 6A is an explanatory diagram of the model update process. The display control unit 13 of the terminal device 10 displays a display screen 900 received from the image management server 40 on the display 106a. The display screen 900 includes a target image 1100 and text 1200.

[0096] The input receiving unit 12 of the terminal device 10 receives, from the microphone 109b, voice information indicating the dialogues Q1, A1, Q2, and A2 between the data provider M1 and the data provider M2 as input information input by the data provider to the displayed display screen 900. It is preferable that the data providers M1 and M2 have a wealth of knowledge about the business, including tacit knowledge. By updating the tacit knowledge model 4004 through such dialogues between the data providers M1 and M2, the user can obtain useful tacit knowledge comments.

[0097] The identification unit 44 identifies the target image 1100, which is the portion of the display screen 900 excluding the text 1200.

[0098] Then, the determination unit 43 determines the relevance between the caption comment acquired from the caption model 4003 using the target image 1100 and the dialogues Q1, A1, Q2, and A2.

[0099] The update unit 46 updates the tacit knowledge model 4004 by treating the comments judged to have low relevance among the dialogues Q1, A1, Q2, and A2 as tacit knowledge comments and using them as learning data together with the target image 1100, etc., and updates the caption model 4003 by treating the comments judged to have high relevance as caption comments and using them as learning data together with the target image 1100.

[0100] Therefore, the tacit knowledge model 4004 learns the correspondence between the target image 1100 and the dialogues Q1, A1, Q2, and A2. Features are extracted from the target image 1100 using several feature extraction models suitable for images, such as a CNN (Convolutional Neural Network). The features represent, for example, what is depicted where, or the work being done in the image. Therefore, the tacit knowledge model 4004 can learn the correspondence between the image features and the dialogues Q1, A1, Q2, and A2.

[0101] FIG. 6(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106 a , and the display screen 900 includes an image 1110 and text 1210 .

[0102] The input receiving unit 12 of the terminal device 10 receives, via the microphone 109b, voice information indicating the questions Q11 and Q12 from the user M3 as input information input by the user to the displayed display screen 900.

[0103] The identification unit 44 identifies the image 1110 that does not include the text 1210 as the target image.

[0104] The text information generation unit 45 uses the image 1110 to acquire tacit knowledge comments based on the tacit knowledge model 4004. The tacit knowledge model 4004 extracts features from the image 1110, determines that the features of the image 1110 in FIG. 6(b) are similar to the image 1110 at the time of update, and can identify dialogues Q1, A1, Q2, and A2 related to the image 1110. The dialogues Q1, A1, Q2, and A2 become tacit knowledge comments.

[0105] In addition, the text information generation unit 45 generates text information related to answers A11 and A12 to questions Q11 and Q12, respectively, based on the large-scale language model 4005, using tacit knowledge comments (dialogues Q1, A1, Q2, A2) and questions Q11 and Q12, etc.

[0106] The display control unit 13 of the terminal device 10 displays the text information relating to the answers A11 and A12 received from the image management server 40 on the display 106a.

[0107] 7 is another explanatory diagram displayed by the terminal device 10 in the model update process and text information generation process according to this embodiment. In FIG. 7, a case where a question sentence is not used in the model update and text information generation will be described.

[0108] Fig. 7(a) is an explanatory diagram of the model update process. Fig. 7(a) shows an example in which the tacit knowledge model 4004 is updated not based on a conversation between data providers, but based on the voice information and partial image of one data provider.

[0109] The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a, and the display screen 900 includes a first image 1100A and a second image 1100B.

[0110] The input receiving unit 12 of the terminal device 10 receives, from the keyboard 110a, character information indicating the comments C1 to C4 by the data provider M4 as input information input by the data provider to the displayed display screen 900.

[0111] In addition, the input receiving unit 12 receives, from the mouse 110b, operation information indicating an operation by the data provider M4 to identify the partial image 1100B1 in the second image 1100B as input information input by the data provider M4 to the displayed display screen 900.

[0112] The specifying unit 44 may specify the partial image 1100B1 as the target image, or may specify the first image 1100A or the second image 1100B as the target image.

[0113] Then, the determination unit 43 determines the degree of association between the caption comment acquired from the caption model 4003 using the target image and the comments C1 to C4.

[0114] The update unit 46 updates the tacit knowledge model 4004 by treating the comments determined to have low relevance among the comments C1 to C4 as tacit knowledge comments and using them as learning data together with the partial image 1100B1, etc., and updates the caption model 4003 by treating the comments determined to have high relevance as caption comments and using them as learning data together with the partial image 1100B1.

[0115] Therefore, the tacit knowledge model 4004 learns the correspondence between the partial image 1100B1 and the comments C1 to C4. Features are extracted from the partial image 1100B1 using several feature extraction models suitable for images, such as CNN. The features represent, for example, what is depicted where, or the work being done. Therefore, the tacit knowledge model 4004 can learn the correspondence between the image features and the comments C1 to C4.

[0116] 7(b) is an explanatory diagram of the text information generation process. The display control unit 13 of the terminal device 10 displays the display screen 900 received from the image management server 40 on the display 106a. The display screen 900 includes an image 1110.

[0117] User M5 does not make any input to the displayed display screen 900, and the input accepting unit 12 does not accept any input information that the user inputs to the displayed display screen 900. The identifying unit 44 identifies image 1110, which is the entire display screen 900, as the target image.

[0118] When user M5 performs an operation to identify partial image 1100B1 on display screen 900, input receiving unit 12 receives, as input information, operation information indicating the operation to identify the partial image from mouse 110b. In this case, identification unit 44 identifies the partial image on display screen 900 as the target image in accordance with the operation information.

[0119] The text information generation unit 45 uses the partial image 1100B1 to acquire tacit knowledge comments based on the tacit knowledge model 4004. The tacit knowledge model 4004 determines that the features of the image 1110B1 in FIG. 7(b) are similar to the image 1110B1 at the time of update, and can identify comments C1 to C4 related to the image 1110B1. The tacit knowledge model 4004 extracts the comments C1 to C4 as tacit knowledge comments.

[0120] The text information generation unit 45 uses tacit knowledge comments and the like to generate text information related to the comments C11 to C14 based on the large-scale language model 4005. The text information generation unit 45 may generate text information using preset fixed questions when no question sentence is input, instead of using a method that does not use any questions at all.

[0121] The display control unit 13 of the terminal device 10 displays the text information relating to the comments C11 to C14 received from the image management server 40 on the display 106a.

[0122] <Action or Processing> Next, a method of displaying inspection information and image information on one screen will be described as an example of processing based on these. In other words, the tacit knowledge model 4004 is not used.

[0123] FIG. 8 is a sequence diagram showing an example of a process for generating screen information in which inspection information and image information are arranged as a process based on both the inspection information and the image information.

[0124] S1: The user inputs a login operation into the terminal device 10. This login is to log in to the property information management server 20. The input acceptance unit 12 of the terminal device 10 accepts the login operation. Any existing login method may be used. Here, it is assumed that the login is successful.

[0125] The user logs in to the property information management server 20 and then logs in to the image management server 40, but may also log in to the image management server 40 first and then log in to the property information management server 20.

[0126] S2: If the login is successful, the transmitting / receiving unit 11 of the terminal device 10 transmits a request for the property specification screen 200 to the property information management server 20.

[0127] S3: The transmitting / receiving unit 21 of the property information management server 20 receives the request for the property specification screen 200. The screen generation unit 22 generates the property specification screen 200, and the transmitting / receiving unit 21 transmits screen information of the property specification screen 200 to the terminal device 10.

[0128] S4: The transmitter / receiver 11 of the terminal device 10 receives screen information of the property specification screen 200. The display control unit 13 displays the property specification screen 200 (see FIG. 9). The user inputs property identification information (for example, V0001) on the displayed property specification screen 200. The input receiving unit 12 of the terminal device 10 receives the property identification information.

[0129] S5: The transmitting / receiving unit 11 of the terminal device 10 transmits a request for property management information to the property information management server 20 by specifying the property identification information.

[0130] S6: The transmitting / receiving unit 21 of the property information management server 20 receives the request for property management information, and the storage / reading unit 29 searches the property information management DB 2001 for the property identification information. The property management information is information about the property associated with the property identification information. The screen generation unit 22 of the property information management server 20 generates a property management screen 210 that displays the property management information, and the transmitting / receiving unit 21 transmits the screen information of the property management screen 210 to the terminal device 10.

[0131] Furthermore, in response to a request for property management information, the transmitter / receiver 21 transmits an image request program to the terminal device 10 so that the terminal device 10 can acquire image information. The image request program is, for example, a Web application. This Web application is installed on the property information management server 20 by the operator of the image management server 40 with the permission of the operator of the property information management server 20. Alternatively, a URL containing the image request program may be transmitted to the terminal device 10. Since the Web application acquires image information from the image management server 40, it has the function of connecting to the image management server 40 and requesting and displaying image information.

[0132] S7: The transmitter / receiver 11 of the terminal device 10 receives the screen information of the property management screen 210 and the image request program. The display control unit 13 displays the property management screen 210 (see FIG. 10). This displays the property management information. The user performs an operation to request image information of the property (presses the image acquisition button 213) in response to the displayed property management information. The input accepting unit 12 of the terminal device 10 accepts the operation to request image information of the property. Image information of the property is image information of items placed in the property generated as a virtual space. The items are composed of 3D model shape information. Note that since the property has already been identified, a request for image information of the property may be sent to the image management server 40 without any operation by the user.

[0133] Here, the property management screen 210 includes a first display area 214 that displays information about the item obtained from the property information management server 20, and a second display area 215 that displays image information about the item obtained from the image management server 40. At step S7, the property management information is displayed in the first display area 214, but nothing is displayed in the second display area 215.

[0134] S8: If the user is not logged in to the image management server 40, the user inputs a login operation into the terminal device 10. This login operation is for the image management server 40. The input acceptance unit 12 of the terminal device 10 accepts the login operation. Any existing login method may be used. Here, it is assumed that the login is successful. Furthermore, the user's login operation may be omitted, for example, by using single sign-on or the like.

[0135] S9: The terminal device 10 executes an image request program to request image information. This causes the transmission / reception unit 11 to specify the property identification information of the property selected by the user and transmit a request for image information of the property to the image management server 40. The transmission / reception unit 11 may transmit the URL of the property information management server 20 to the image management server 40 so that the terminal device 10 can be redirected to the property information management server 20. The image information of the property is an image of an object placed in a property formed as a virtual space. Since the object is configured using 3D model shape information, the terminal device 10 generates a planar image by projecting the three-dimensional model shape of the object into two dimensions. The user can view any object by changing the viewpoint. The transmission / reception unit 11 may also transmit property management information acquired from the property information management server 20 to the image management server 40. The image request program receives the property management information, for example, as a URL parameter, from a web application connected to the property information management server 20.

[0136] S10: The transmitting / receiving unit 41 of the image management server 40 receives a request for image information of the property. The storage / reading unit 49 searches the image information management DB 4001 using the property identification information and obtains image information of each item. The processing unit 47 requests the screen generation unit 42 to generate a screen including the image information of the property. The screen generation unit 42 generates a screen corresponding to the second display area 215 in which the image information of each item is arranged on one screen. The transmitting / receiving unit 41 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The image information of each item included in the screen information is image information in which all items included in the property are arranged on the property, and the user can freely change the viewpoint.

[0137] S11: The transmitter / receiver 11 of the terminal device 10 receives screen information of the screen corresponding to the second display area 215, and the display control unit 13 displays the property display screen 220 including the first display area 214 and the second display area 215 (see FIG. 11). At the stage of step S11, only image information of each item is displayed in the second display area 215, but the first display area 214 displays, for example, a list of items (generated from the property management information) placed in the property, so that the item list and image information of the property are displayed on one screen. The item list is included in the property management information. Next, the user identifies any item from the image information of the property. Once the user identifies the item, inspection information for the item can be requested. The input receiving unit 12 of the terminal device 10 receives an operation to identify the item. The item may be identified, for example, by the coordinates clicked by the user, or model identification information may be identified by these coordinates.

[0138] S12: When the user presses the information display button 225, the transmitting / receiving unit 11 of the terminal device 10 transmits information specifying the item to the image management server 40.

[0139] S13: The transmitting / receiving unit 41 of the image management server 40 receives the information identifying the item. The storage / reading unit 49 identifies the model identification information of the item having the closest location information from the image information management DB 4001 based on the information identifying the item (location information in this case). The transmitting / receiving unit 41 transmits the model identification information to the terminal device 10. Note that if the information identifying the item in step S12 is model identification information, there is no need to search the image information management DB 4001. The image management server 40 transmits the model identification information in order to obtain the latest inspection information from the property information management server 20.

[0140] S14: The transmitting / receiving unit 11 of the terminal device 10 receives the model identification information. For example, the image management server 40 notifies the terminal device 10 of the URL of the property information management server 20 and causes it to be redirected. As a result, the transmitting / receiving unit 11 of the terminal device 10 specifies the model identification information and transmits a request for item inspection information to the property information management server 20.

[0141] S15: The transmitting / receiving unit 21 of the property information management server 20 receives the request for inspection information of the item, and the storage / reading unit 29 searches the property information management DB 2001 for the model identification information to obtain the inspection information. The transmitting / receiving unit 21 transmits the inspection information to the terminal device 10.

[0142] S16: When the transmitting / receiving unit 11 of the terminal device 10 receives the inspection information, it transmits the inspection information to the image management server 40 while specifying the model identification information.

[0143] S17: When the transmitting / receiving unit 41 of the image management server 40 receives the inspection information, the storage / reading unit 49 identifies the item using the model identification information and saves the latest inspection information in the image information management DB 4001. Next, the processing unit 47 associates the image information of the item corresponding to the model identification information with the latest inspection information and requests the screen generation unit 42 to display them. As a process based on the information about the item and the image information, the screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the inspection information and the image information of the item in association with each other. The screen generation unit 42 may perform an update process that adds only the inspection information to the screen corresponding to the second display area 215. The transmitting / receiving unit 41 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10.

[0144] S18: The transmitter / receiver 11 of the terminal device 10 receives image information of the screen information of the screen corresponding to the second display area 215, and the display control unit 13 displays the inspection image screen 230 including the first display area 214 and the second display area 215 (see FIG. 12). In step S18, the first display area 214 displays the item list, as in step S11, and the second display area 215 displays image information of the items and inspection information corresponding to the image information.

[0145] <Screen example> 9 shows an example of a property specification screen 200 for inputting property identification information. The property specification screen 200 has a property identification information input field 201 and a search button 202. When a user inputs property identification information into the property identification information input field 201 and presses the search button 202, a list of room numbers as shown in FIG. 10 is displayed on the property management screen 210.

[0146] FIG. 10 is a diagram showing an example of a property management screen 210. The property management screen 210 includes a first display area 214 that displays information about items acquired from the property information management server 20, and a second display area 215 that displays image information about items acquired from the image management server 40. The first display area 214 is an area other than the second display area 215. The first display area 214 includes a list 211 of room numbers of the property identified by the property identification information. The list 211 of room numbers corresponds to property management information, and includes items and other information included in each property. Depending on the property, the room numbers may not be displayed, and the screen may transition from the property specification screen 200 in FIG. 9 to FIG. 11, where image information about the property is displayed. The user selects the room number for which image information is to be displayed using the mouse cursor 212. When the user presses the image acquisition button 213, a property display screen 220 is displayed.

[0147] The second display area 215 is an area other than the first display area 214, and may be displayed by a program on a Web application, such as an iframe.

[0148] FIG. 11 illustrates an example of a property display screen 220. The property display screen 220 includes a first display area 214 and a second display area 215, and displays both property management information and image information on a single screen. An item list 221 is displayed in the first display area 214 of the property display screen 220. The items in this room are a table, a cylindrical monument, a door, and a storage box. Image information 222 of these items is displayed in the second display area 215 of the property display screen 220. The user can select an item for which inspection information is to be displayed from the image information using the mouse cursor 212. This determines the coordinates of the item as information identifying the item. Alternatively, the user may identify the item from the item list 221. In this case, model identification information is identified at the time of selection. When the user presses the information display button 225, an inspection image screen 230 is displayed. The information display button 225 not only displays inspection information, but also displays text information generated based on tacit knowledge comments, as described below. Furthermore, when the user presses the information update button 226, the tacit knowledge model 4004 is updated.

[0149] 11, the information about the property is displayed as size (floor area) 224. Size (floor area) 224 may be a measured value or may be included in the property management information.

[0150] 11, the terminal device 10 can display on one screen the property management information managed by the property information management server 20 and the image information of the property managed by the image management server 40. The user can check the image information of the property while looking at the property management information.

[0151] FIG. 12 is a diagram showing an example of an inspection image screen 230. The inspection image screen 230 includes a first display area 214 and a second display area 215, and includes inspection information and image information of an item. In FIG. 12, image information of a table 223 selected by a user as an example of an item is displayed in the second display area 215. Furthermore, the second display area 215 displays image information of the table 223 and inspection information 232 corresponding to the image information of the table 223. An example of the inspection information 232 is "Table: Size 0mm*0mm*0mm." In this way, the terminal device 10 can display image information of an item and inspection information 232 corresponding to the image information of the item on a single screen.

[0152] As shown in FIG. 12, the user can check the image information of the item and the inspection information 232 of the item corresponding to the image information of the item on one screen while viewing the property management information.

[0153] As shown in FIG. 13, while image information of an item is displayed in second display area 215, the user can change the item to be displayed in second display area 215. FIG. 13 is a diagram illustrating an example of switching between image information of an item and inspection information displayed in second display area 215. From the state of FIG. 12, the user moves the position of mouse cursor 212 in item list 221 to a storage box and presses it. A message that storage box has been pressed is transmitted to image management server 40, and screen generation unit 42 generates a screen corresponding to second display area 215 that displays the inspection information of the storage box and image information of the item in association with each other. Transmitting / receiving unit 41 transmits screen information of the screen corresponding to second display area 215 to terminal device 10. Therefore, image information of storage box 227 and inspection information 228 are displayed in second display area 215 in FIG. 13.

[0154] In this way, when image information of a first item (table) is displayed, information about the first item (inspection information 232) associated with the image information of the first item is displayed. In this state, when the display is changed from the image information of the first item to image information of a second item (storage box), the display is changed to information about the second item (inspection information 228) associated with the image information of the second item.

[0155] <Example of an image management server obtaining inspection information from a property information management server> In Figure 8, the image management server 40 acquires the inspection information that the terminal device 10 acquired from the property information management server 20, but it is also possible for the image management server 40 to acquire the inspection information directly from the property information management server 20.

[0156] Fig. 14 is a sequence diagram showing an example of a process for generating screen information based on inspection information and image information. Note that Fig. 14 may mainly explain the differences from Fig. 8. First, the processes of steps S1 to S12 may be the same as those in Fig. 8.

[0157] S21: The transmitting / receiving unit 41 of the image management server 40 receives information identifying the item. The storage / reading unit 49 identifies the model identification information of the item having the closest location information from the image information management DB 4001 based on the information identifying the item (location information in this case). The processing unit 47 requests inspection information from the transmitting / receiving unit 41. The transmitting / receiving unit 41 calls the API of the property information management server 20, thereby specifying the model identification information and sending a request for inspection information to the property information management server 20. Note that if the information identifying the item is model identification information, there is no need to search the image information management DB 4001.

[0158] S22: The transmitting / receiving unit 21 of the property information management server 20 receives the request for inspection information. The storage / reading unit 29 searches the property information management DB 2001 for the model identification information and acquires the inspection information. The transmitting / receiving unit 21 transmits the inspection information to the image management server 40. The transmitting / receiving unit 41 acquires the inspection information from the property information management server 20 as a response to the request.

[0159] The subsequent processing may be the same as in Fig. 8. With the processing as in Fig. 14, the processing of changing the connection destination by the terminal device 10 can be reduced, and the time until the inspection image screen 230 is displayed can be shortened.

[0160] <Major effects, etc.> In the image management server of this embodiment, the image management server 40 performs processing based on the inspection information and image information, so the terminal device 10 can display the image information and the second display area 215 of the inspection image screen 230 that includes inspection information corresponding to the image information. The image management server 40 can perform processing based on the inspection information managed by the property information management server 20 and the image information managed by the image management server 40 without adding any processing functions to the property information management server 20.

[0161] In addition, the property information management server 20 may perform some of the processing based on the inspection information managed by the property information management server 20 and the image information managed by the image management server 40, and even in this case, the processing load on the property information management server 20 is reduced compared to when all of the processing based on the inspection information managed by the property information management server 20 and the image information managed by the image management server 40 is performed on the property information management server 20 side.

[0162] Here, the image information and the inspection information corresponding to the image information may be displayed so as not to overlap each other, or may be displayed so as to overlap each other. Furthermore, the first display area 214 and the second display area 215 may be displayed so as not to overlap each other, or may be displayed so as to overlap each other. Furthermore, each of the first display area 214 and the second display area 215 may be divided into a plurality of areas and displayed, or the respective divided areas may be displayed in a mixed manner.

[0163] [Second embodiment] In this embodiment, an image management server 40 will be described which acquires tacit knowledge comments from a tacit knowledge model using inspection information and generates text information based on the tacit knowledge comments.

[0164] In this embodiment, the hardware configuration diagram of FIG. 2 and the functional block diagram of FIG. 3 described in the above embodiment can be used.

[0165] <Action or Processing> <<Learning Phase (Model Update)>> First, the model update process in which the tacit knowledge model 4004 learns data will be described with reference to Fig. 15. Fig. 15 is a sequence diagram showing an example of the model update process. In the description of Fig. 15, differences from Fig. 8 may be mainly explained. The processes of steps S31 to S40 may be the same as those in Fig. 8.

[0166] S41: The transmitter / receiver 11 of the terminal device 10 receives screen information of a screen corresponding to the second display area 215, and the display control unit 13 displays a property display screen 220 including the first display area 214 and the second display area 215 (see FIG. 11). For example, a list 221 of items placed in the property and image information 222 of the property are displayed. Next, the user identifies an item from the image information. Once the user identifies the item, inspection information for the item can be requested. The input receiving unit 12 of the terminal device 10 receives an operation to identify the item. The item may be identified, for example, by the coordinates clicked by the user, or the coordinates may identify model identification information. The user also inputs a comment about the item into the terminal device 10, such as a comment (text information or voice) as described in FIGS. 6 and 7. The comment may be referred to as input information. This comment may be a tacit knowledge comment. The comment may also include a caption comment describing the item.

[0167] S42: The transmitting / receiving unit 11 of the terminal device 10 transmits the information for identifying the item and the input information to the image management server 40.

[0168] The subsequent processing in steps S43 to S46 may be the same as in FIG.

[0169] S47: The transmitting / receiving unit 41 of the image management server 40 receives the inspection information. The storage / reading unit 49 identifies the property using the model identification information and saves the latest inspection information in the image information management DB 4001. The judgment unit 43 acquires the caption comment identified by the model identification information from the caption model 4003 and determines the degree of relevance between the caption comment and the comment included in the input information received in step S42. The judgment unit 43 may determine the degree of relevance between the acquired caption comment and all of the comments included in the input information received in step S42, or may divide the comment included in the input information received in step S42 into multiple comments and determine the degree of relevance between each of the divided comments and the acquired caption comment.

[0170] S48: The update unit 46 updates the caption model 4003 by associating the comment determined to have a high degree of relevance in step S47 as a caption comment with the model identification information. The update unit 46 also updates the tacit knowledge model 4004 using the comment and inspection information determined to have a low degree of relevance in step S47 and the image information of the related item (identified in step S42) as learning data. That is, the correspondence between the image information of the item and the comment and inspection information is learned. Features are extracted from the image information of the item using several feature extraction models suitable for images, such as CNN. The features represent, for example, what is depicted where, or the work being performed. Therefore, the tacit knowledge model 4004 can learn the correspondence between the features of the image information of the item and the comment and inspection information.

[0171] It should be noted that both the comments and the inspection information are not necessarily required, and the tacit knowledge model 4004 can be updated if at least one of them is present.

[0172] 15, the image management server 40 acquires the inspection information from the terminal device 10, but it may acquire the inspection information from the property information management server 20 as in FIG.

[0173] <<Example of a learning phase screen>> The screens displayed by the terminal device 10 in the learning phase are similar to those shown in Figures 9 to 11, but the user can input input information in Figure 11. Also, information corresponding to Figure 12, which displays inspection information, is displayed in the inference phase, which will be described next.

[0174] 16 is a diagram showing an example of a property display screen 220 displayed by the terminal device 10. The property display screen 220 in FIG. 16 includes a first display area 214 and a second display area 215. The first display area 214 displays a size (floor area) 224 as information about the property, and the second display area 215 displays image information of a table 223 and input information 241 corresponding to the image information of the table 223, such as "This table has an unstable center of gravity, so it is best not to place anything weighing more than 50 kg on it." The image management server 40 can update the tacit knowledge model 4004 using such input information 241 and inspection information. The size (floor area) 224 as information about the property can be used as a caption comment.

[0175] <Inference phase (generation of text information)> Next, the process of generating text information using the tacit knowledge model 4004 will be described with reference to Fig. 17. Fig. 17 is a sequence diagram of an example of the process of generating text information. Note that the explanation of Fig. 17 may mainly focus on the differences from Fig. 8. The processes of steps S31 to S46 may be the same as those of Fig. 8. However, in step S41, the user inputs a question about an item (see Fig. 18).

[0176] S51: The transmitting / receiving unit 41 of the image management server 40 receives inspection information. The storage / reading unit 49 identifies the item using the model identification information and saves the latest inspection information in the image information management DB 4001. The processing unit 47 requests the text information generation unit 45 to generate text information. The text information generation unit 45 acquires tacit knowledge comments corresponding to the image information of the item from the tacit knowledge model 4004. The tacit knowledge model 4004 can extract features of the image information of the item and identify at least one of comments and inspection information corresponding to these features. The tacit knowledge model 4004 extracts at least one of comments and inspection information as tacit knowledge comments. S52: Next, the text information generation unit 45 acquires text information created by the large-scale language model using the tacit knowledge comment, the input information (question sentence), and the inspection information. The large-scale language model 4005 can use these to generate more detailed text information. The text information generation unit 45 may convert speech information included in the input information into character information, and the text information generated by the text information generation unit 45 may be either speech information or character information.

[0177] The text information generation unit 45 may generate text information without using any inspection information or question text. Alternatively, the text information generation unit 45 may generate fixed questions within the system and use the fixed questions. In this case, the questions are not visible to the user. Alternatively, the text information generation unit 45 may generate fixed questions within the system, display the fixed questions on the display unit, allow the user to select one, and use the selected question.

[0178] In this way, although inspection information is not required, more detailed information about the item can be obtained by using the inspection information to generate text information from the large-scale language model 4005. For example, if the inspection information includes the degree of damage to the item, text information can be generated that includes an appropriate response according to the degree of damage.

[0179] S53: The processing unit 47 associates the image information and text information of the item corresponding to the model identification information, and requests the screen generation unit 42 to display them. The screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the image information of the item and text information corresponding to the image information of the item. The screen generation unit 42 may perform update processing to add only the text information to the screen corresponding to the second display area 215. The transmission / reception unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmission / reception unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0180] S54: The display control unit 13 of the terminal device 10 displays the inspection image screen 230 including the first display area 214 and the second display area 215 including text information (see FIG. 19). Alternatively, the conversion unit 15 converts the received text information into audio information, and the audio control unit 14 plays the converted text information on the speaker 109a. Furthermore, if the received text information is audio information, the text information is played on the speaker 109a, or the conversion unit 15 converts the received text information into character information and displays the converted text information on the display 106a.

[0181] In FIG. 17, the image management server 40 acquires the inspection information from the terminal device 10, but it may also acquire the inspection information from the property information management server 20 as in FIG.

[0182] <<Example of an inference phase screen>> The screens displayed by the terminal device 10 in the inference phase are similar to those shown in FIGS. 9 to 11, but in FIG. 11 the user can input input information.

[0183] FIG. 18 is a diagram showing an example of a property display screen 220 in the inference phase. The property display screen 220 includes a first display area 214 and a second display area 215. FIG. 18 has the same configuration as FIG. 11, but the user has input a question as input information. The second display area 215 displays image information of a table 223 and input information (question) 234 corresponding to the image information of the table 223. For example, the input information (question) 234 in FIG. 18 is "There is a scratch on the table. What should I do?" Along with this input information 234, the user presses the information display button 225 to request the generation of text information using the tacit knowledge model.

[0184] FIG. 19 is a diagram showing an example of text information displayed on the inspection image screen 230. The inspection image screen 230 includes a first display area 214 and a second display area 215. The second display area 215 displays image information of a table 223 and text information 235 corresponding to the image information of the table 223. This text information 235 reads, "The scratch is less than 1 mm deep, so repair it with paint. If it is more than 1 mm, polish it." The text information 235 is generated by the large-scale language model 4005 from the image information of the item, the inspection information, and the question. For example, if a scratch is detected in the image information of the item, a tacit knowledge comment regarding the scratch on the item is extracted. This tacit knowledge comment, the question regarding the scratch, and the inspection information specifying the current state of the scratch are input into the large-scale language model 4005, allowing appropriate text information corresponding to the current scratch to be generated.

[0185] The text information 235 is not the inspection information itself, but includes at least one of text information generated by an implicit knowledge model that has learned the inspection information and text information generated based on the inspection information by the large-scale language model 4005. Furthermore, the text information 235 can be said to be the processing result of processing based on the inspection information and image information.

[0186] <Effects of generating text information using inspection information> The effect of generating text information using inspection information as in this embodiment will be described.

[0187] 1. Comparative Example 1 (Using a General Large-Scale Language Model) Q: A user asks, "How do I repair the cracks?" A: You can use tape or filler.

[0188] 2. Comparative Example 2 (Learning Image Information) Learning Phase Q: While viewing the image, the user asks, "How can I repair the crack?" Comments: Use tape for larger cracks and filler for smaller cracks. Inference phase Q: Only image display A: There are large cracks and small cracks, so we recommend using tape for the former and filler for the latter.

[0189] 3. This embodiment (image information only) The learning phase is the same as in Comparative Example 2. Inference phase Q: Only image display Inspection information obtained: There is a crack in the tape at the corner A: There are large cracks and small cracks, so we recommend using tape for the former and filler for the latter, but please be careful when applying tape to corners as it may cause cracks. In other words, the effect of obtaining inspection information during inference is "Please be careful when applying tape to corners as it may cause cracks."

[0190] 4. This embodiment (image information and input information) Learning Phase Q: While viewing the image, the user asks, "How can I repair the crack?" Comments: Use tape for larger cracks and filler for smaller cracks. Obtained inspection information: The wide crack spans the corner Inference phase Q: Only image display A: There are large cracks and small cracks, so we recommend using tape for the former and filler for the latter, but please be careful when applying tape to corners as there is a risk of cracks occurring. In other words, the effect of learning the inspection information is that "When applying tape to corners, there is a risk of cracks occurring so please be careful when applying tape."

[0191] <Multimodal> Several examples of combinations of input information and tacit knowledge comments will be described. While the above-mentioned model is premised on a large-scale language model, this embodiment can utilize a multimodal model that inputs multiple data formats (images, text, gestures, etc.) and outputs them in a predetermined data format. - When the input information is a string and content other than text information is generated as a tacit knowledge comment Enter a string and generate an image Enter a string and generate a video Input a string and generate audio Enter a string and generate a 3D model - When the input information contains both text and non-text, and the text information is generated as a tacit knowledge comment Generate text information by inputting images and text strings Generate text information by inputting a 3D model and a string of characters Generate text information by inputting voice and text strings - When the input information contains both text and non-text information, and content other than text information is generated as a tacit knowledge comment Generate an image by inputting an image and a string Generate a video by inputting a video and a string Generate a 3D model by inputting a 3D model and text. Generate speech by inputting voice and text <Major Effects> The image management server of this embodiment updates the tacit knowledge model with the inspection information and image information as processing based on the inspection information and image information, so the terminal device 10 can display the image information and the tacit knowledge comment corresponding to the image information corresponding to the acquired inspection information. Even if the inspection information cannot be acquired when generating text information, the tacit knowledge model can output the tacit knowledge comment that takes the inspection information into consideration based on the acquired image information in accordance with the image information.

[0192] [Other application examples] The present invention is not limited to the specifically disclosed embodiment described above, and various modifications and changes are possible without departing from the scope of the claims. Note that the image management server 40 described in this embodiment is only one example, and it goes without saying that there are various system configuration examples depending on the application and purpose.

[0193] For example, in this embodiment, an example has been described in which a tacit knowledge model from an industry such as civil engineering or construction answers questions, but the tacit knowledge model may be used in any industry where tacit knowledge is effective, such as medicine, dentistry, or investment decisions.

[0194] Furthermore, in this embodiment, the large-scale language model 4005 generates text information based on tacit knowledge comments, but the tacit knowledge comments may be used as text information without using the large-scale language model 4005.

[0195] Furthermore, the tacit knowledge model 4004 may be one that receives image information and inspection information as input, outputs the input information, and learns tacit knowledge comments. In other words, different forms of information, such as images and text, may be input.

[0196] Furthermore, in this embodiment, the information processing system 100 has been described as being of a client-server type, but the functions of the image management server 40 may be installed as an application on the terminal device 10. In other words, the user may be able to use the functions of this embodiment in a standalone format.

[0197] Furthermore, the configuration examples shown in Figure 3 and elsewhere are divided according to main functions to make it easier to understand the processing by the image management server 40. The method of dividing the processing units or the names of the processing units does not limit the present invention. The processing by the image management server 40 can also be divided into more processing units depending on the processing content. Furthermore, it can also be divided so that one processing unit includes even more processes.

[0198] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above.

[0199] The devices described in the example are merely one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, image management server 40 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communications link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0200] Furthermore, the image management server 40 can combine the disclosed processing steps in various ways. The elements of the image management server 40 may be integrated into a single device or may be separated into multiple devices. Furthermore, each process performed by the image management server 40 may be performed by the terminal device 10.

[0201] <Aspect> [Aspect 1] An information processing system including a first server that manages information about an item, a second server that manages image information about the item, and a terminal device that can communicate with the first server and the second server, The terminal device a display control unit that displays a display screen including information about the item received from the first server and image information about the item received from the second server, The second server a processing unit that executes a process of associating the image information of the item with the information about the item or generation information generated based on the information about the item, based on the information about the item and image information about the item transmitted by the first server; The display control unit of the terminal device displaying, on the display screen, image information of the item and information or generated information about the item corresponding to the image information, received from the second server; Information processing system. [Aspect 2] The display control unit of the terminal device a first display area that displays information about the item received from the first server; displaying the display screen including the image information of the item received from the second server and a second display area that displays information about the item or the generation information according to the image information; 2. The information processing system according to claim 1. [Aspect 3] The second server storing the information about the item transmitted by the first server in a storage unit in association with image information about the item; 3. The information processing system according to claim 1 or 2. [Aspect 4] the processing unit requests information about the item from the first server, and obtains the information about the item from the first server in response to the request; 4. The information processing system according to any one of aspects 1 to 3. [Aspect 5] The second server a model that has learned the correspondence between image information of the item and information about the item; The processing unit acquiring, as the generation information, text information generated by the model based on image information of the item selected by the terminal device; 2. An information processing system according to claim 1. [Aspect 6] The second server a model that has learned correspondence between image information of the item, information about the item, and input information received from a terminal device; The processing unit acquiring, as the generation information, text information generated by the model based on image information of the item selected by the terminal device; 2. An information processing system according to claim 1. [Aspect 7] The second server an update unit that updates the model by learning correspondence between image information of the article and information about the article; An information processing system according to embodiment 5. [Aspect 8] The second server an update unit that updates the model by learning correspondence between image information of the item, information about the item, and input information received from a terminal device; An information processing system according to embodiment 6. [Aspect 9] The image information of the article is a two-dimensional projection of a three-dimensional model shape of the article, and the image information of the article can be displayed by changing the viewpoint. 9. The information processing system according to any one of aspects 1 to 8. [Aspect 10] The information about the item is information input by a user after inspecting the condition of the item. 10. The information processing system according to any one of aspects 1 to 9. [Aspect 11] The terminal device In a state where the display control unit displays image information of the item and information about the item corresponding to the image information or the generated information on the display screen, an input receiving unit for receiving a selection of an item other than the item for which the image information is displayed; The processing unit executes a process of associating the image information of the other item with the information about the other item or generated information generated based on the information about the other item, based on the information about the other item and the image information of the other item transmitted by the first server; The display control unit displaying, on the display screen, the image information of the other item received from the second server, and the information about the other item corresponding to the image information or the generated information; 10. The information processing system according to any one of aspects 1 to 9. [Explanation of symbols]

[0202] 10 Terminal Equipment 20 Property information management server 40 Image Management Server 100 Information Processing Systems [Prior art documents] [Patent documents]

[0203] [Patent Document 1] Patent No. 6954410

Claims

1. An information processing system including a first server that manages information about an item, a second server that manages image information about the item, and a terminal device that can communicate with the first server and the second server, The terminal device a display control unit that displays a display screen including information about the item received from the first server and image information about the item received from the second server; The second server a processing unit that executes a process of associating the image information of the item with the information about the item or generation information generated based on the information about the item, based on the information about the item and image information about the item transmitted by the first server; The display control unit of the terminal device displaying, on the display screen, the image information of the item and the information about the item corresponding to the image information or the generated information, which are received from the second server; Information processing system.

2. The display control unit of the terminal device a first display area that displays information about the item received from the first server; displaying the display screen including the image information of the item received from the second server and a second display area that displays information about the item or the generation information according to the image information; The information processing system according to claim 1 .

3. The second server storing the information about the item transmitted by the first server in a storage unit in association with image information about the item; 3. The information processing system according to claim 1 or 2.

4. the processing unit requests information about the item from the first server, and obtains the information about the item from the first server in response to the request; The information processing system according to claim 1 .

5. The second server a model that has learned the correspondence between image information of the item and information about the item; The processing unit acquiring, as the generation information, text information generated by the model based on image information of the item selected by the terminal device; The information processing system according to claim 1 .

6. The second server a model that has learned correspondence between image information of the item, information about the item, and input information received from a terminal device; The processing unit acquiring, as the generation information, text information generated by the model based on image information of the item selected by the terminal device; The information processing system according to claim 1 .

7. The second server an update unit that updates the model by learning correspondence between image information of the article and information about the article; The information processing system according to claim 5 .

8. The second server an update unit that updates the model by learning correspondence between image information of the item, information about the item, and input information received from a terminal device; The information processing system according to claim 6.

9. The image information of the article is a two-dimensional projection of a three-dimensional model shape of the article, and the image information of the article can be displayed by changing the viewpoint. The information processing system according to claim 1 .

10. The information about the item is information input by a user after inspecting the condition of the item. The information processing system according to claim 1 .

11. The terminal device In a state where the display control unit displays image information of the item and information about the item corresponding to the image information or the generated information on the display screen, an input receiving unit for receiving a selection of an item other than the item for which the image information is displayed; The processing unit executes a process of associating the image information of the other item with the information about the other item or generation information generated based on the information about the other item, based on the information about the other item and the image information of the other item transmitted by the first server; The display control unit displaying, on the display screen, the image information of the other item received from the second server, and the information about the other item corresponding to the image information or the generated information; The information processing system according to claim 1 .

12. a second server in a system including a first server that manages information about an item, a second server that manages image information about the item, and a terminal device that is capable of communicating with the first server and the second server and that displays a display screen including information about the item received from the first server and image information about the item received from the second server, a processing unit that executes a process of associating the image information of the item with the information about the item or generation information generated based on the information about the item, based on the information about the item and the image information of the item transmitted by the first server; a transmitting unit that transmits to the terminal device screen information for displaying image information of the item and information about the item corresponding to the image information or the generated information on the display screen. Second server.

13. a first server in a system including a first server that manages information about an item, a second server that manages image information about the item, and a terminal device that is capable of communicating with the first server and the second server and that displays a display screen including information about the item received from the first server and image information about the item received from the second server, a transmitting unit that transmits information about the item to the terminal device to display on the display screen; The transmitting unit further transmits information about the item to the terminal device in response to a request from the terminal device that has received a request from the second server, or transmits information about the item to the second server in response to a request from the second server, in order to display image information about the item and information about the item corresponding to the image information about the item or generated information generated based on the information about the item on the display screen. The first server.

14. An information processing method performed by a second server in a system including a first server that manages information about an item, a second server that manages image information about the item, and a terminal device that is capable of communicating with the first server and the second server and that displays a display screen including information about the item received from the first server and image information about the item received from the second server, the method comprising: a process of executing a process of associating the image information of the item with the information about the item or generation information generated based on the information about the item, based on the information about the item and the image information of the item transmitted by the first server; transmitting screen information for displaying, on the display screen, image information of the item and information about the item corresponding to the image information of the item or the generated information to the terminal device; An information processing method that performs the above.

15. In a system including a first server that manages information about an item, a second server that manages image information about the item, and a terminal device that is capable of communicating with the first server and the second server and that displays a display screen including information about the item received from the first server and image information about the item received from the second server, executing a process of associating information about the item transmitted by the first server with generated information generated based on image information about the item; transmitting screen information for displaying, on the display screen, image information of the item and information about the item corresponding to the image information of the item or the generated information, to the terminal device; A program that executes the following.

Images