Information processing device, information processing method, program, and information processing system

Abstract

To easily associate positional information, which indicates the three-dimensional position of an object in three-dimensional data, with information related to the object. [Solution] The system includes a creation unit that creates a display screen including a first display area that displays a two-dimensional image aligned to three-dimensional information and a second display area that displays information about an article; a processing unit that associates position information indicating the three-dimensional position of the article in the three-dimensional information corresponding to a two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area; and a storage control unit that stores the information associated by the processing unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, a program, and an information processing system.

Background Art

[0002] Conventionally, there has been known a so-called digital twin technology that acquires three-dimensional information of the real space as digital data by a laser scanner, LiDAR (Light Detection and Ranging), photogrammetry, etc., and reproduces it on a computer. In recent years, the acquired digital data has been used for facility management of buildings and management of design information in construction.

[0003] Patent Document 1 describes an architectural information integrated management system that enables seamless access to appropriate information between three-dimensional CAD data and architectural-related information.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to the conventional technology, it has been necessary to associate position information indicating the position of an article (equipment, spare parts, etc.) in a building, which is a three-dimensional position in three-dimensional information, with information regarding the article (information used for management). Further, there has been a problem that it is time-consuming to perform such association for each of a plurality of articles.

[0005] The present invention has been made in view of the above, and an object thereof is to easily associate position information indicating the three-dimensional position of an article in three-dimensional information with information regarding the article.

Means for Solving the Problems

[0006] The system includes a creation unit that creates a display screen including a first display area that displays a two-dimensional image aligned to three-dimensional information and a second display area that displays information about an item; a processing unit that associates position information indicating the three-dimensional position of the item in the three-dimensional information corresponding to a two-dimensional position specified in the two-dimensional image with information about the item displayed in the second display area; and a storage control unit that stores the information associated by the processing unit. [Effects of the Invention]

[0007] According to the present invention, positional information indicating the three-dimensional position of an article in three-dimensional information can be easily associated with information related to the article. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is an overall configuration diagram showing an example of an information processing system according to the first embodiment. [Figure 2] Figure 2 is a hardware configuration diagram showing an example of a terminal device and management server according to the first embodiment. [Figure 3] Figure 3 is a functional block diagram showing an example of an information processing system according to the first embodiment. [Figure 4] Figure 4 shows an example of the display screen of the 3D viewer. [Figure 5] Figure 5 shows another example of the 3D viewer's display screen. [Figure 6] Figure 6 shows an example of equipment information. [Figure 7] Figure 7 shows an example of a display screen created by the creation unit. [Figure 8] Figure 8 shows another example of a display screen created by the creation unit. [Figure 9] Figure 9 shows another example of a display screen created by the creation unit. [Figure 10] Figure 10 shows another example of a display screen created by the creation unit. [Figure 11]Figure 11 shows another example of equipment information. [Figure 12] Figure 12 is a flowchart showing an example of the procedure for the mapping process according to the first embodiment. [Figure 13] Figure 13 is a functional block diagram showing an example of an information processing system according to the second embodiment. [Figure 14] Figure 14 shows another example of a display screen created by the creation unit. [Figure 15] Figure 15 is a flowchart showing an example of the procedure for the mapping process according to the second embodiment. [Modes for carrying out the invention]

[0009] (First Embodiment) Figure 1 is an overall configuration diagram showing an example of an information processing system 1 according to the first embodiment. The information processing system 1 of this embodiment is constructed by a terminal device 10 and a management server 40.

[0010] The management server 40 is an example of an information processing device that performs one or more information processing operations on three-dimensional information such as three-dimensional point clouds and mesh objects.

[0011] Here, a three-dimensional point cloud is a collection of coordinate points in the X, Y, and Z directions that correspond to measurement points on the surface of an object when the space in which the object is located is measured using a laser scanner (LS) or the like. Each coordinate point is denoted, for example, as (1, 3, 5). Color information may also be attached to each coordinate point, and the values ​​of the color components R (Red), G (Green), and B (Blue) of each coordinate point may be attached as color information.

[0012] Three-dimensional point clouds are sometimes called point clouds. Point cloud data is data that can be handled by computers and other devices as a collection of coordinate points in a virtual three-dimensional space.

[0013] In the above, an example of measuring a three-dimensional point cloud using LS was shown, but the three-dimensional point cloud may also be measured using other optical measurement means or mechanical measurement means. Examples of other optical measurement means include a method using a stereo camera and a method using Visual SLAM (Simultaneous Localization And Mapping).

[0014] Note that the three-dimensional information may include a digital model (3D model) of a building or facilities inside a building used in BIM (Building Information Modeling) etc. and a 3D-CAD (Computer Aided Design) model. Also, facilities (equipment, installations, devices, etc.) are an example of articles.

[0015] The terminal device 10 and the management server 40 can communicate via the communication network 20. The communication network 20 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), etc. The communication network 20 may include not only wired communication but also networks by wireless communication such as 3G (3rd Generation), WiMAX (Worldwide interoperability for Microwave Access), LTE (Long Term Evolution), 5G (5th Generation). Also, the terminal device 10 can communicate by a short-range communication technology such as NFC (Near Field Communication) (registered trademark).

[0016] <Hardware Configuration> FIG. 2 is a hardware configuration diagram showing an example of the terminal device 10 and the management server 40 according to the first embodiment. Each hardware configuration of the terminal device 10 is indicated by a reference numeral in the 100 series. Each hardware configuration of the management server 40 is indicated by a reference numeral in the 400 series.

[0017] The terminal device 10 includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HDD (Hard Disk Drive) 104, HDD controller 105, display I / F (interface) 106, communication I / F 107, sensor I / F 108, sound input / output I / F 109, input I / F 110, media I / F 111, DVD (Digital Versatile Disk)-RW drive 112, and bus line 113.

[0018] Of these components, the CPU 101 controls the operation of the entire terminal device 10. The ROM 102 stores the program used to drive the CPU 101. The RAM 103 is used as the work area for the CPU 101. The HDD 104 stores various data such as programs. The HDD controller 105 controls the reading or writing of various data to the HDD 104 according to the control of the CPU 101.

[0019] The display interface 106 is for connecting a display 106a that displays various information such as cursors, menus, windows, text, or images. The communication interface 107 is for data communication using the communication network 20. The sensor interface 108 is for connecting various sensors.

[0020] The audio input / output interface 109 is an interface for connecting speaker 109a and microphone 109b. The input interface 110 is an interface for connecting input devices such as keyboard 110a and mouse 110b. Here, keyboard 110a is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. Mouse 110b is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor.

[0021] The media interface 111 controls the reading or writing (storage) of data to or from the recording medium 111a, such as flash memory.

[0022] The DVD-RW drive 112 controls the reading or writing of various data to a DVD-RW 112a, which is an example of a removable recording medium. The terminal device 10 may also be configured to control the reading or writing (storage) of data to an external PC (Personal Computer) or external device connected via wired or wireless means such as Wi-Fi (Wireless Fidelity).

[0023] Furthermore, the management server 40 is equipped with a CPU 401, ROM 402, RAM 403, HDD 404, HDD controller 405, display I / F 406, communication I / F 407, sensor I / F 408, sound input / output I / F 409, input I / F 410, media I / F 411, DVD-RW drive 412, and bus line 413. Since these have the same configuration as described above (CPU 101, ROM 102, RAM 103, HDD 104, HDD controller 105, display I / F 106, communication I / F 107, sensor I / F 108, sound input / output I / F 109, input I / F 110, media I / F 111, DVD-RW drive 112, and bus line 113), their descriptions will be omitted.

[0024] The DVD-RW drive 112 (412) may be replaced with a DVD-R drive or the like. Furthermore, the terminal device 10 and the management server 40 may each be constructed using a single computer, or they may be constructed using multiple computers, each arbitrarily assigned to a different component (function, means, or storage unit).

[0025] Figure 3 is a functional block diagram showing an example of an information processing system 1 according to the first embodiment.

[0026] <Functional Configuration of Terminal Devices> As shown in Figure 3, the terminal device 10 includes a transmitting / receiving unit 11, a receiving unit 12, a display control unit 14, and a storage / reading unit 19. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 101 following a program deployed from the HDD 104 onto the RAM 103. The terminal device 10 also has a storage unit 1000 constructed from the RAM 103 and HDD 104 shown in Figure 2.

[0027] (Functional configuration of each terminal device) Next, we will describe each component of the terminal device 10.

[0028] The transmitting / receiving unit 11 is an example of a receiving means and is implemented by commands from the CPU 101 shown in Figure 2 and the communication I / F 107, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 20.

[0029] The reception unit 12 is an example of a reception means and is mainly implemented by commands from the CPU 101 shown in Figure 2, as well as the keyboard 110a and mouse 110b, and accepts various inputs from the user.

[0030] The display control unit 14 is an example of a display control means, and is implemented by instructions from the CPU 101 shown in Figure 2, and causes various images and screens to be displayed on the display 106a, which is an example of a display unit.

[0031] The terminal device 10 displays three-dimensional information using, for example, an application (3D viewer) that visualizes three-dimensional information. The 3D viewer can be a web application used in a web browser, a PC application that runs on a PC, or a smartphone application that runs on a smartphone.

[0032] The three-dimensional information can be uploaded to a cloud service, stored on a terminal device 10 such as a PC or smartphone, etc. The terminal device 10 can visualize the three-dimensional information on a 3D viewer regardless of the type of 3D viewer or the location of the three-dimensional information. The three-dimensional information can be in any format as long as it can be divided into constituent elements, such as a mesh or point cloud.

[0033] Figure 4 shows an example of the display screen of the 3D viewer. The information processing system 1 uses the 3D viewer to visualize three-dimensional information and image information of the real space that was captured in advance, thereby realizing a so-called digital twin. Figure 4 shows a display screen 300 that includes a first display area 310 that displays a two-dimensional image (arbitrary viewpoint image) of the inside of the machine room viewed from a certain viewpoint.

[0034] The image information in this embodiment is information about a two-dimensional image (such as a 360-degree image, a photograph, or a texture image generated from a photograph) that is aligned with three-dimensional information. This two-dimensional image is aligned with three-dimensional information by, for example, embedding the coordinate values ​​of the three-dimensional information as metadata in each pixel. Therefore, if a two-dimensional position (pixel) is specified in the two-dimensional image, the three-dimensional position in the three-dimensional information corresponding to the specified two-dimensional position can be determined.

[0035] Figure 5 shows another example of the 3D viewer's display screen. In this example, a first display area 310 is shown that displays a two-dimensional image (overhead view) of the entire machine room shown in Figure 4. The user can view the visualized three-dimensional information by switching between displaying an arbitrary viewpoint image like Figure 4 and an overhead view image like Figure 5. Furthermore, the user can change the viewpoint of the arbitrary viewpoint image and the angle of the overhead view image by inputting via the reception unit 12, allowing them to check the contents of the three-dimensional information from various angles and viewpoints. In this way, the 3D viewer enables the viewing and overhead display of images from any shooting point within the building, as well as walkthroughs in three-dimensional space.

[0036] The storage / reading unit 19 is an example of a storage control unit. The storage / reading unit 19 is implemented by instructions from the CPU 101 shown in Figure 2, as well as the HDD controller 105, media I / F 111, DVD-RW drive 112, and an external PC or external device. The storage / reading unit 19 stores various data in the storage unit 1000, recording media 111a, DVD-RW 112a, and the external PC or external device. The storage / reading unit 19 also reads various data from the storage unit 1000, recording media 111a, DVD-RW 112a, and the external PC or external device. The storage unit 1000, recording media 111a, DVD-RW 112a, and the external PC or external device are examples of storage units.

[0037] <Management Server Functional Configuration> As shown in Figure 3, the management server 40 includes a transmitting / receiving unit 41, a processing unit 43, a creation unit 47, and a storage / reading unit 49. Each of these units is a function or means of functioning, realized by any of the components shown in Figure 2 operating according to instructions from the CPU 401 following a program deployed from the HDD 404 onto the RAM 403. The management server 40 also has a storage unit 4000 constructed from the HDD 404 shown in Figure 2. The storage unit 4000 is an example of a storage means.

[0038] (Configuration of each function of the management server) Next, we will describe the various components of the management server 40. The management server 40 may be configured to distribute its functions across multiple computers. Furthermore, although we will describe the management server 40 as a server computer located in a cloud environment, it may also be a server located in an on-premises environment.

[0039] The transmitting / receiving unit 41 is an example of a transmission means and is implemented by commands from the CPU 401 shown in Figure 2 and the communication I / F 407, and transmits and receives various data (or information) with other terminals, devices, or systems via the communication network 20.

[0040] Furthermore, the transmitting / receiving unit 41 receives equipment information. The received equipment information is stored in the item information management DB (Database) 4003 via the storage / reading unit 49. Here, equipment information is information about equipment within a building. Equipment information includes, for example, the equipment's ID (Identification), the location where the equipment is installed, the equipment's name, and maintenance dates. Note that equipment information is an example of information about items.

[0041] Figure 6 shows an example of equipment information. As shown in Figure 6, equipment information includes various pieces of information about the equipment (equipment ID, room code, equipment name, inspection date, etc.). Buildings have many pieces of equipment, and there are set times for statutory inspections and maintenance. The "Last Inspection Date" and "Next Inspection Date" information in Figure 6 are used to manage the inspection and maintenance schedules for each piece of equipment.

[0042] Furthermore, the "three-dimensional position" in Figure 6 represents the coordinates (location information) of the equipment's installation location, for example, in the format (X, Y, Z). The user can use this information to identify the location of equipment that requires maintenance. This embodiment enables the management of each piece of equipment within a building by easily specifying and storing location information for each of many pieces of equipment.

[0043] The equipment information may also be generated by the management server 40. For example, an administrator can add, edit, etc., equipment information by operating the keyboard 410a and mouse 410b. The added or edited equipment information is stored in the item information management DB 4003 via the storage / reading unit 49. The equipment information file may be in any format as long as it can describe the information for each item. For example, Excel format or CSV format may be used as the file format.

[0044] The transmitting / receiving unit 41 may receive equipment information in file format or receive it from an external system's database using an API (Application Programming Interface). Furthermore, the item names for the equipment information shown in Figure 6 are examples, and different items may be used depending on the circumstances of the building where the equipment is installed.

[0045] The processing unit 43 is implemented by instructions from the CPU 401 shown in Figure 2, and performs various processes, including the process of associating location information with equipment information, which will be described later.

[0046] The creation unit 47 is implemented by instructions from the CPU 401 shown in Figure 2, and creates a display screen 300 for display on a 3D viewer or the like.

[0047] The storage / reading unit 49 is an example of a storage control unit. The storage / reading unit 49 is implemented by instructions from the CPU 401 shown in Figure 2, as well as the HDD controller 405, media I / F 411, DVD-RW drive 412, and an external PC or external device. The storage / reading unit 49 stores various data in the storage unit 4000, recording media 411a, DVD-RW 412a, and the external PC or external device. The storage / reading unit 49 also reads various data from the storage unit 4000, recording media 411a, DVD-RW 412a, and the external PC or external device. The storage unit 4000, recording media 411a, DVD-RW 412a, and the external PC or external device are examples of storage units.

[0048] The memory unit 4000 contains an image information management DB 4001, a three-dimensional information management DB 4002, and an item information management DB 4003. The memory unit 4000 may be implemented using multiple servers. For example, the image information management DB 4001 and the three-dimensional information management DB 4002 may be configured on a first server, while the item information management DB 4003 may be configured on a second server.

[0049] Image information management DB4001 stores and manages image information, three-dimensional information management DB4002 stores and manages three-dimensional information, and item information management DB4003 stores and manages equipment information (information related to items).

[0050] Next, using Figures 7 to 12, we will explain the process of matching the screen created by the creation unit 47 with the processing unit 43.

[0051] Figure 7 shows an example of a display screen 300 created by the creation unit 47. The display screen 300 in Figure 7 includes a first display area 310 that displays a two-dimensional image aligned to three-dimensional information, and a second display area 320 that displays equipment information. For example, while viewing a two-dimensional image in the first display area 310 as in Figures 4 and 5, the user can add the second display area 320 to the display screen 300 by selecting the equipment information display function of the 3D viewer. Note that the first display area 310 is an example of a first display area, and the second display area 320 is an example of a second display area.

[0052] In the second display area 320 of Figure 7, only the equipment name, model number, and last inspection date are displayed from the information contained in the equipment information. By limiting the number of items displayed in this way, equipment information for multiple pieces of equipment can be displayed in the second display area 320.

[0053] Figure 8 shows another example of the display screen 300 created by the creation unit 47. The difference from Figure 7 is that the equipment information of the equipment selected from multiple pieces of equipment is displayed in the second display area 320, and additional display items are also displayed. The user can use the reception unit 12 to select the desired equipment from the multiple pieces of equipment displayed in the second display area 320 of Figure 7, thereby displaying the equipment information of the selected equipment in the second display area 320.

[0054] In Figure 8, additional items such as the installation date and inspection-related laws and regulations (the names of the laws and regulations that form the basis of the inspection) are displayed in the second display area 320. The user can edit the contents of each item displayed in the second display area 320 using the reception unit 12.

[0055] Furthermore, a location selection button 321 is displayed in the second display area 320 of Figure 8. The location selection button 321 is displayed when location information is not associated with the equipment information of the selected equipment. For example, if the equipment information is received from spreadsheet or equipment ledger management software, location information is not associated in this way. That is, the equipment information does not include a three-dimensional location item. In this embodiment, in such cases, the location information and equipment information are associated as follows.

[0056] The user can display the position pointer 311 in the first display area 310 by pressing the position selection button 321 using the reception unit 12 (mouse 110b, etc.). The user can also specify a two-dimensional position by moving the position pointer 311 to a desired two-dimensional position using the reception unit 12 and pressing the position selection button 321.

[0057] The processing unit 43 associates location information, which indicates the three-dimensional position of the equipment in the three-dimensional information, with equipment information, corresponding to the specified two-dimensional position. The storage / reading unit 49 stores the information associated by the processing unit 43 in the storage unit 4000, etc. The associated information includes the coordinates of the installation location added as the three-dimensional position of the equipment information, equipment information written to the data of the position indicated by the location information in the three-dimensional information, and link information to the equipment information file written to the relevant data.

[0058] Furthermore, if location information is associated with the equipment information of the selected equipment, the creation unit 47 creates a display screen 300 that displays a two-dimensional image of the area around the two-dimensional location corresponding to that location information in the first display area 310. This allows the user to check images of the selected equipment and its surroundings. In this case, a location change button may be displayed in the second display area 320 instead of the location selection button 321. The user can change (edit) the existing location information by pressing the location change button to display and move the location pointer 311, and then pressing the location change button again.

[0059] Furthermore, if location information is not associated with the equipment information, the creation unit 47 may create a display screen 300 that displays a two-dimensional image of a location close to the equipment's location in the first display area 310 using the room code or other information from the equipment information. This allows the user to easily specify the two-dimensional location of the equipment. Note that information such as the room code is assumed to be stored in the three-dimensional information when the three-dimensional information is captured, for example.

[0060] Furthermore, if location information is not associated with the equipment information and room codes are not included, the processing unit 43 may, along with the association process, add a code indicating the room containing the two-dimensional location specified by the user as a room code to the equipment information. This allows the user to add room codes to the equipment information by specifying the two-dimensional location of the equipment.

[0061] Figure 9 shows another example of a display screen created by the creation unit 47. The difference from Figure 7 is that a location icon 322 is displayed in the equipment information displayed in the second display area 320. The location icon 322 indicates that the equipment information is associated with the location information of the three-dimensional information. In the example in Figure 9, a location icon 322 is displayed in the equipment information of the air conditioning equipment, indicating that the equipment information of the air conditioning equipment is already associated with the location information. On the other hand, a location icon 322 is not displayed in the equipment information of the piping, indicating that the equipment information of the piping is not yet associated with the location information.

[0062] In this way, the presence or absence of the location icon 322 allows users to easily recognize whether or not the equipment information is associated with location information. The location icon 322 is an example of an identification image that indicates that location information and equipment information are associated.

[0063] Figure 10 shows another example of a display screen created by the creation unit 47. The difference from Figure 9 is that a thumbnail image 323 is displayed with the equipment information displayed in the second display area 320. The processing unit 43 extracts a partial image of the two-dimensional image at a two-dimensional position specified by the user and associates it with the equipment information. The creation unit 47 then displays the extracted partial image as a thumbnail image 323 in the second display area 320.

[0064] For equipment information that is not associated with location information, the processing unit 43 displays, for example, a pre-prepared image for the equipment name as a thumbnail image 323 in the second display area 320. In this case, the thumbnail image 323 may be a text image containing the equipment name, an icon image schematically representing the equipment, or an icon image containing a general photograph of the equipment.

[0065] The range from which the processing unit 43 extracts a partial image from the two-dimensional image may be either a pre-set range including a two-dimensional position specified by the user, or a range selected by the user via the receiving unit 12.

[0066] In this way, the display of thumbnail images 323 allows the user to easily select the desired equipment from the second display area 320. Note that thumbnail images 323 are examples of partial images of a two-dimensional image at a two-dimensional position.

[0067] Figure 11 shows another example of equipment information. The difference from Figure 6 is that a section for partial images has been added to the equipment information. In the example in Figure 11, the file name of the partial image associated with the equipment information has been added to the section for partial images. Note that instead of the file name of the partial image, link information to the partial image may also be added to the equipment information.

[0068] Figure 12 is a flowchart showing an example of the procedure for the mapping process according to the first embodiment.

[0069] First, the transmitting / receiving unit 41 receives equipment information (step S100), and the creation unit 47 creates a display screen 300 including a first display area 310 and a second display area 320 (step S101). At this time, the second display area 320 displays multiple pieces of equipment information received by the transmitting / receiving unit 41. The created display screen 300 is then displayed on the display 106a by the display control unit 14. Thereafter, the created display screen 300 will be similarly displayed on the display 106a by the display control unit 14.

[0070] Next, the user selects the desired equipment from the second display area 320 (step S102), and the creation unit 47 creates a display screen 300 that displays detailed information of the selected equipment in the second display area 320 (step S103).

[0071] If the selected equipment information is associated with location information (Step S104: Yes), the association process is terminated. On the other hand, if the selected equipment information is not associated with location information (Step S104: No), the user specifies the two-dimensional location of the equipment (Step S105). Specifically, the user presses the location specification button 321 to display the location pointer 311, moves the location pointer 311 to the desired location, and then presses the location specification button 321 to specify the two-dimensional location.

[0072] Then, the processing unit 43 associates the location information with the equipment information based on the specified two-dimensional position, and the storage / reading unit 49 stores the associated information in the storage unit 4000, etc. (step S106).

[0073] Furthermore, in the above, the functions of the management server 40 in Figure 3 may be integrated into the terminal device 10, and each of the processes of the management server 40 described above may also be executed by the terminal device 10.

[0074] Thus, according to this embodiment, by using the first display area 310 and the second display area 320, position information indicating the three-dimensional position in three-dimensional information can be easily associated with equipment information.

[0075] (Second Embodiment) The second embodiment uses a trained AI (Artificial Intelligence) model to display candidate two-dimensional positions of the equipment in the second display area 320. In the following description of the second embodiment, the description of parts that are the same as those of the first embodiment will be omitted, and the parts that differ from the first embodiment will be described.

[0076] Figure 13 is a functional block diagram showing an example of an information processing system 1 according to the second embodiment. The difference from Figure 3 is that a trained model 4004 is further constructed in the storage unit 4000 of the management server 40. Note that if the storage unit 4000 is implemented by multiple servers, the trained model 4004 may be constructed on any of the servers. For example, the trained model 4004 may be constructed on the first server or on the second server.

[0077] The trained model 4004 in Figure 13 is an AI model (machine learning model) that has been trained using training data that includes equipment information and two-dimensional images. The two-dimensional images used in machine learning are, for example, images of the equipment indicated in the equipment information.

[0078] Here, machine learning is a technique for enabling computers to acquire human-like learning abilities. It refers to a technique in which a computer autonomously generates algorithms necessary for data identification and other judgments from pre-inputted training data, and then applies these algorithms to new data to make predictions. The learning method for machine learning can be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or a combination of these learning methods; the learning method for machine learning is not restricted.

[0079] The processing unit 43 searches for an image of the equipment indicated by the equipment information from a two-dimensional image aligned to three-dimensional information, based on the trained model 4004 and the equipment information selected by the user and displayed in the second display area 320, and obtains the position of the searched image as a candidate for the two-dimensional position.

[0080] The creation unit 47 creates a display screen 300 by superimposing an instruction image showing candidate two-dimensional positions acquired by the processing unit 43 onto a two-dimensional image aligned with three-dimensional information, and displays it in the first display area 310.

[0081] Figure 14 shows another example of a display screen created by the creation unit 47. The difference from Figure 8 is that a candidate pointer 312 (an example of an instruction image showing a candidate for a two-dimensional position) is displayed in the first display area 310, and a candidate position determination button 324 has been added to the second display area 320.

[0082] The user can confirm the position of the candidate pointer 312 and press the candidate position determination button 324 to determine the position of the candidate pointer 312 as the two-dimensional position of the equipment displayed in the second display area 320. In this embodiment, the position information and equipment information can be easily associated by determining the two-dimensional position of the equipment based on the trained model 4004.

[0083] Furthermore, the user can move the position of the candidate pointer 312 using the reception unit 12 and then press the position selection button 321 to specify the new position of the candidate pointer 312 as a two-dimensional position. This allows for easy correction of the two-dimensional position determined based on the trained model 4004.

[0084] Alternatively, instead of displaying the candidate position determination button 324 in the second display area 320, the processing unit 43 may use the two-dimensional position obtained based on the trained model 4004 to associate the position information with the equipment information. In this case as well, the user can move the position of the candidate pointer 312 and press the position specification button 321 to specify the position of the moved candidate pointer 312 as the two-dimensional position.

[0085] Furthermore, the training data used for the trained model 4004 may also include three-dimensional information. By training the model using three-dimensional information in addition to two-dimensional images, the accuracy of image retrieval can be improved.

[0086] In this embodiment as well, the position icon 322 shown in Figure 9 can be displayed in the second display area 320. In this case, the image of the position icon 322 indicating that it is associated based on a two-dimensional position specified by the user and the image of the position icon 322 indicating that it is associated based on a two-dimensional position determined based on the trained model 4004 may be different. For example, the two may be identification images with different shapes or colors. This allows the user to easily identify the equipment information associated with the position information based on the trained model 4004, and to check or correct the position information as needed.

[0087] The location icon 322, which indicates that the location is associated with a two-dimensional position specified by the user, is an example of a first identification image. The location icon 322, which indicates that the location is associated with a two-dimensional position determined based on the trained model 4004, is an example of a second identification image. Alternatively, instead of using different images for the location icon 322, when displaying the equipment information associated with the location information based on the trained model 4004 in the second display area 320, a tag such as "AI detection result" may be added.

[0088] Figure 15 is a flowchart showing an example of the procedure for the mapping process according to the second embodiment. The processes in steps S200 to S202 and S207 in Figure 15 are the same as the processes in steps S100 to S102 and S106 in Figure 12, so their explanation is omitted.

[0089] The processing unit 43 acquires candidate two-dimensional positions using the trained model 4004 (step S203). The creation unit 47 then creates a display screen 300 that displays an instruction image indicating the candidate two-dimensional positions in the first display area 310 (step S204).

[0090] If the user decides to use one of the acquired two-dimensional position candidates as the two-dimensional position of the equipment (Step S205: Yes), the process proceeds to Step S207. On the other hand, if the user does not decide to use one of the acquired two-dimensional position candidates as the two-dimensional position of the equipment (Step S205: No), the user changes (moves) the position of the two-dimensional position candidate to specify the two-dimensional position of the equipment (Step S206).

[0091] Thus, according to this embodiment, by determining the two-dimensional position of the equipment based on the trained model 4004, the effort required of the user to specify the two-dimensional position can be reduced.

[0092] The programs executed by the information processing devices of each embodiment described above are provided as installable or executable files recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).

[0093] Furthermore, the program executed by the information processing device of each embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the information processing device of each embodiment may be provided or distributed via a network such as the Internet.

[0094] Furthermore, the programs for each embodiment may be pre-installed and provided in ROM or the like.

[0095] The program executed in the information processing device of each embodiment has a modular configuration that includes the above-mentioned parts (processing unit 43, creation unit 47, storage / reading unit 49, etc.), and in actual hardware, the CPU (processor) reads the program from the recording medium and executes it, thereby loading the above-mentioned parts onto the main memory and generating the program.

[0096] Each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each function described above.

[0097] Furthermore, the apparatus described in each embodiment represents only one of several computing environments for carrying out each embodiment disclosed herein.

[0098] In one embodiment, the management server 40 includes multiple computing devices, such as a server cluster. These computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein. Similarly, the terminal device 10 may include multiple computing devices configured to communicate with each other.

[0099] Although various embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0100] Examples of the present invention are as follows: <1> A creation unit that creates a display screen including a first display area for displaying a two-dimensional image aligned to three-dimensional information, and a second display area for displaying information about an item. A processing unit that associates position information indicating the three-dimensional position of the article in the three-dimensional information corresponding to the two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area, A storage control unit that stores the information associated by the aforementioned processing unit, This is an information processing device equipped with [a specific feature / feature]. <2> The creation unit creates the display screen which displays information about at least one of the multiple articles in the second display area, The processing unit associates the position information with the information relating to the article whose two-dimensional position is specified and which is displayed in the second display area. <1> This is the information processing device described above. <3> The creation unit creates the display screen which displays an instruction image indicating a candidate for the two-dimensional position in the first display area, based on a trained model trained using training data including information about the item and a two-dimensional image, and information about the item displayed in the second display area. <2> This is the information processing device described above. <4> The processing unit further associates a partial image of the two-dimensional image at the two-dimensional position with information about the article. <1> ~ <3> It is an information processing device described in any one of the following. <5> The creation unit creates the display screen which displays an identification image in the second display area that indicates that the location information and the information about the item are associated. <1> ~ <4> It is an information processing device described in any one of the following. <6> The aforementioned creation unit, A display screen is created that displays in the second display area a first identification image indicating that the position information and the information about the item are associated based on the two-dimensional position specified by the user. A display screen is created that displays a second identification image, different from the first identification image, in the second display area, indicating that the position information and the information about the article are associated based on the two-dimensional position determined by a trained model trained using training data including the information about the article and the three-dimensional information. <1> ~ <5> It is an information processing device described in any one of the following. <7> A display control unit that displays the aforementioned display screen on the display unit, A receiving unit that receives a specification of the two-dimensional position in the two-dimensional image displayed in the first display area, The aforementioned comprising <1> ~ <6> It is an information processing device described in any one of the following. <8> An information processing method performed by an information processing device, A creation process for creating a display screen that includes a first display area for displaying a two-dimensional image aligned to three-dimensional information, and a second display area for displaying information about an item. A processing step that associates position information indicating a three-dimensional position in the three-dimensional information corresponding to a two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area. A storage control step for storing the information associated by the processing step, This is an information processing method that includes [something]. <9> Computers, A means for creating a display screen that includes a first display area for displaying a two-dimensional image aligned to three-dimensional information, and a second display area for displaying information about an item. A processing means that associates position information indicating a three-dimensional position in the three-dimensional information corresponding to a two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area. A storage control means for storing the information associated by the processing means, This is a program that makes it function as such. <10> An information processing system comprising an information processing device and a terminal device capable of communicating with the information processing device, A creation unit that creates a display screen including a first display area for displaying a two-dimensional image aligned to three-dimensional information, and a second display area for displaying information about an item. A processing unit that associates position information indicating a three-dimensional position in the three-dimensional information corresponding to a two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area, A storage control unit that stores the information associated by the aforementioned processing unit, It is an information processing system equipped with [the following features]. [Explanation of Symbols]

[0101] 1. Information Processing System 10 Terminal devices 12 Reception Department 14 Display Control Unit 20 Communication Networks 40 Management Server 43 Processing Unit 47 Creation Department 49 Memory / readout section 300 display screen 310 1st display area 311 Position Pointer 312 Candidate pointers 320 2nd display area 321 Position selection button 322 Location Icons 323 thumbnail images 324 Candidate Position Confirmation Button [Prior art documents] [Patent Documents]

[0102] [Patent Document 1] Patent No. 5895037

Claims

1. A creation unit that creates a display screen including a first display area for displaying a two-dimensional image aligned to three-dimensional information, and a second display area for displaying information about an item. A processing unit that associates position information indicating the three-dimensional position of the article in the three-dimensional information corresponding to the two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area, A storage control unit that stores the information associated by the aforementioned processing unit, An information processing device equipped with the following features.

2. The creation unit creates the display screen which displays information about at least one of the multiple articles in the second display area. The information processing apparatus according to claim 1, wherein the processing unit associates the position information with information relating to the article displayed in the second display area and whose two-dimensional position has been specified.

3. The information processing apparatus according to claim 2, wherein the creation unit creates a display screen in which an instruction image indicating a candidate for the two-dimensional position is displayed in the first display area, based on a trained model trained using training data including information about the article and a two-dimensional image, and information about the article displayed in the second display area.

4. The information processing apparatus according to claim 1, wherein the processing unit further associates a partial image of the two-dimensional image at the two-dimensional position with information relating to the article.

5. The information processing apparatus according to claim 1, wherein the creation unit creates a display screen in which an identification image indicating that the location information and the information relating to the article are associated is displayed in the second display area.

6. The aforementioned creation unit, A display screen is created that displays a first identification image in the second display area, indicating that the position information and information about the item are associated based on the two-dimensional position specified by the user. The information processing apparatus according to claim 1, which creates a display screen that displays in a second display area a second identification image different from the first identification image, indicating that the position information and the information about the article are associated based on the two-dimensional position determined by a trained model trained using training data including the information about the article and the three-dimensional information.

7. A display control unit that displays the aforementioned display screen on the display unit, A receiving unit that receives a specification of the two-dimensional position in the two-dimensional image displayed in the first display area, An information processing apparatus according to any one of claims 1 to 6, comprising:

8. An information processing method performed by an information processing device, A creation process for creating a display screen that includes a first display area for displaying a two-dimensional image aligned to three-dimensional information, and a second display area for displaying information about an item. A processing step that associates position information indicating a three-dimensional position in the three-dimensional information corresponding to a two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area. A storage control step for storing the information associated by the processing step, Information processing methods including

9. Computers, A means for creating a display screen that includes a first display area for displaying a two-dimensional image aligned to three-dimensional information, and a second display area for displaying information about an item. A processing means that associates position information indicating a three-dimensional position in the three-dimensional information corresponding to a two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area. A storage control means for storing the information associated by the processing means, A program that makes it function as such.

10. An information processing system comprising an information processing device and a terminal device capable of communicating with the information processing device, A creation unit that creates a display screen including a first display area for displaying a two-dimensional image aligned to three-dimensional information, and a second display area for displaying information about an item. A processing unit that associates position information indicating a three-dimensional position in the three-dimensional information corresponding to a two-dimensional position specified in the two-dimensional image with information about the article displayed in the second display area, A storage control unit that stores the information associated by the aforementioned processing unit, An information processing system equipped with the following features.

Images