Information processing apparatus, three-dimensional reconstruction system, and information processing method

Abstract

An information processing device (100) that performs three-dimensional reconstruction is provided with a job estimation unit (110) and a metadata generation unit (120). A work estimation unit (110) estimates work performed by a worker on a device in a target period on the basis of at least one of machine device data indicating the state of the device in the target period and sensor data that is data acquired by a sensor provided in a terminal operated by the worker in the target period. On the basis of the result of the estimation work, the part and the state of the part of the equipment that is the work target of the worker within the target period are estimated. A metadata generation unit (120) generates, as metadata, data indicating each of the estimated part and the state of the estimated part, and applies the generated metadata to reconstruction data, which is data that indicates the part, is acquired by the terminal during the target period, and is used when performing a three-dimensional reconstruction process.

Description

Technical Field

[0001] The invention relates to an information processing device, a three-dimensional reconstruction system and an information processing method. Background Art

[0002] There is a method for three-dimensionally reconstructing an object using image data or point group data obtained by observing the object from multiple viewpoints. As a specific example, the method is photogrammetry. As a specific example, when three-dimensionally reconstructing an object using only image data obtained by photographing the object using this method, the image data are connected to each other based on the similarity between feature points, and the posture of the camera when the image data was obtained is estimated to reconstruct the object in three dimensions.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2018 / 150515 Pamphlet Summary of the invention

[0006] Problems to be solved by the invention

[0007] When performing three-dimensional reconstruction on large machinery and equipment, etc., there are sometimes multiple parts with similar features. In this case, as a specific example, in the three-dimensional reconstruction of part A, the three-dimensional reconstruction is performed in a state where the data of part A and the data of part B having a shape similar to that of part A are mixed. According to the technology disclosed in patent document 1, it is possible to roughly determine the range shown in the image data based on the position where the image was acquired and the inclination of the terminal where the image was acquired. However, it is not possible to estimate in more detail which part of which machine is captured in the image only by the position where the image was acquired and the inclination of the terminal where the image was acquired. Therefore, in the technology disclosed in patent document 1, there is a problem that the accuracy of three-dimensional reconstruction is sometimes low.

[0008] In addition, when the machine device has a movable part, data representing different states of the movable part is sometimes taken into the three-dimensional reconstruction result as noise. As a specific example, when the machine device is an elevator, the landing door is equivalent to the movable part. In this case, when three-dimensional reconstruction is performed using both data corresponding to the state of the landing door being open and data corresponding to the state of the landing door being closed, one data is taken in as noise. However, Patent Document 1 does not disclose a technology for preventing such noise from being taken into the three-dimensional reconstruction result. Therefore, in the technology disclosed in Patent Document 1, there is a problem that the accuracy of three-dimensional reconstruction is sometimes reduced.

[0009] An object of the present invention is to improve the accuracy of three-dimensional reconstruction in three-dimensional reconstruction of large-scale machinery and equipment or the like.

[0010] Means for solving problems

[0011] The information processing device of the present invention performs three-dimensional reconstruction of at least a part of the device based on data obtained from at least a part of the observation device, wherein the information processing device has:

[0012] a work estimating unit that estimates work performed by the operator on the equipment during the target period based on at least one of machine equipment data indicating a state of the equipment during the target period and sensor data, which is data acquired during the target period by a sensor provided in a terminal operated by the operator, and estimates a part of the equipment that was a work object of the operator during the target period and the state of the part based on a result of the estimated work; and

[0013] A metadata generating unit generates data representing an estimated part and a state of the estimated part as metadata, and assigns the generated metadata to reconstruction data, which represents the part and is acquired by the terminal during the object period and used when performing three-dimensional reconstruction processing.

[0014] Effects of the Invention

[0015] According to the present invention, metadata is added to reconstruction data. Here, the metadata respectively indicates the part and the state of the part. Therefore, the part indicated by the metadata can be used to determine the part indicated by the reconstruction data, and the state of the part indicated by the metadata can be used to prevent the introduction of noise. Therefore, according to the present invention, in the three-dimensional reconstruction of large-scale machinery and equipment, etc., the accuracy of three-dimensional reconstruction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a diagram showing a configuration example of a three-dimensional reconstruction system 90 according to the first embodiment.

[0017] Figure 2 This is a diagram for explaining metadata according to the first embodiment.

[0018] Figure 3 This is a diagram showing a hardware configuration example of the information processing device 100 according to the first embodiment.

[0019] Figure 4 This is a flowchart showing the operation of the data acquisition device 400 according to the first embodiment.

[0020] Figure 5 This is a flowchart showing the operation of information processing device 100 according to Embodiment 1.

[0021] Figure 6This is a diagram for explaining the process of generating metadata in the first embodiment.

[0022] Figure 7 This is a flowchart showing the operation of information processing device 100 according to Embodiment 1.

[0023] Figure 8 This is a diagram for explaining the processing of the information processing device 100 according to the first embodiment.

[0024] Fig. 9 1 is a diagram showing a configuration example of a three-dimensional reconstruction system 90 according to a modification of the first embodiment.

[0025] Fig.10 This is a diagram showing a hardware configuration example of an information processing device 100 according to a modification of the first embodiment.

[0026] Fig.11 This is a diagram showing a configuration example of a three-dimensional reconstruction system 90 according to the second embodiment.

[0027] Fig.12 This is a flowchart showing the operation of information processing device 100 according to the second embodiment.

[0028] Fig.13 This is a flowchart showing the operation of information processing device 100 according to the second embodiment. DETAILED DESCRIPTION

[0029] In the description of the embodiments and the accompanying drawings, the same reference numerals are used to identify the same elements and corresponding elements. The description of the elements identified with the same reference numerals is appropriately omitted or simplified. The arrows in the figures mainly indicate data flows or processing flows. In addition, "device" or "unit" may be appropriately replaced with "method", "circuit", "process", "procedure", "processing" or "circuit".

[0030] Implementation method 1.

[0031] Hereinafter, the present embodiment will be described in detail with reference to the drawings.

[0032] ***Description of the structure***

[0033] Figure 1 The following is a configuration example of a three-dimensional reconstruction system 90 according to the present embodiment. As shown in the figure, the three-dimensional reconstruction system 90 includes an information processing device 100, a portable terminal 200, a device 300, a data acquisition device 400, a reconstruction data DB (Database) 190, a device data DB 410, a sensor data DB 420, and a reconstruction data DB 430. The devices and DBs constituting the three-dimensional reconstruction system 90 may be appropriately configured as one body.

[0034] The information processing device 100 includes a work estimation unit 110, a metadata generation unit 120, a grouping unit 130, a three-dimensional reconstruction unit 140, an integration unit 150, and a reconstruction result output unit 160. The information processing device 100 is a device that performs three-dimensional reconstruction of at least a part of the equipment based on data obtained by observing at least a part of the equipment.

[0035] The work estimation unit 110 estimates the work of the operator based on the data stored in the equipment data DB 410 and the sensor data DB 420. The work estimation unit 110 estimates the work performed by the operator on the equipment during the target period based on at least one of the equipment data and the sensor data. In addition, the work estimation unit 110 estimates the parts of the equipment that are the work targets of the operator during the target period and the states of the parts based on the results of the estimated work. The target period may be any period.

[0036] The equipment data is data indicating the state of each device constituting the equipment during the target period, and as a specific example, is composed of control signal 310 and sensor signal 320. The equipment data may be data indicating the state of each device estimated based on control signal 310 and sensor signal 320.

[0037] The sensor data is data acquired during the target period by a sensor (excluding the reconstruction data acquisition sensor 240 ) included in a terminal operated by an operator. As a specific example, the terminal is the portable terminal 200 .

[0038] The metadata generator 120 generates metadata based on the work estimated by the work estimation unit 110, and assigns the generated metadata to the reconstruction data stored in the reconstruction data DB 430. At this time, the metadata generator 120 generates data indicating the part estimated by the work estimation unit 110 and the state of the part estimated by the work estimation unit 110 as metadata, and assigns the generated metadata to the reconstruction data. The metadata generator 120 may also assign metadata to a plurality of reconstruction data.

[0039] The reconstruction data is data acquired by the terminal during the target period, data acquired by the operator operating the portable terminal 200 during the operation, data representing each part of the machine, and data directly used when performing the three-dimensional reconstruction process. As a specific example, the reconstruction data is image data or point group data. In addition, the reconstruction data is data corresponding to the operation of the operator.

[0040] The grouping unit 130 classifies each reconstruction data according to metadata assigned to each reconstruction data. At this time, the grouping unit 130 groups the plurality of reconstruction data according to the parts corresponding to the plurality of reconstruction data and the states of the parts.

[0041] The three-dimensional reconstruction unit 140 performs three-dimensional reconstruction on each of the categories classified by the grouping unit 130 using the reconstruction data belonging to each category.

[0042] The integration unit 150 generates data representing frames of the machine using the design data 191, and integrates the result of the three-dimensional reconstruction performed by the three-dimensional reconstruction unit 140 and the generated data representing the frames. At this time, the integration unit 150 generates data representing frames corresponding to the machine having each device, and attaches the result of the three-dimensional reconstruction process performed using the grouped reconstruction data to the generated data representing the frames.

[0043] Design data 191 is data indicating design contents related to the 3D reconstruction object, and as a specific example, is a design drawing indicating the structure of each device or data indicating the design value of each device. As a specific example, the design value is a value indicating at least one of the lifting process, the number of floors, and the distance between floors. Design data 191 is used to grasp the entire contents of the equipment to be 3D reconstructed.

[0044] The reconstruction result output unit 160 outputs data indicating the result of executing the three-dimensional reconstruction. The reconstruction result output unit 160 may receive the reference state definition 192 and output data indicating the result according to the state of each part based on the received reference state definition 192.

[0045] The reference state definition 192 is information indicating the definition of each state of a portion of the device whose state changes.

[0046] The portable terminal 200 has an air pressure sensor 210, an acceleration sensor 220, an angular velocity sensor 230, a reconstruction data acquisition sensor 240, and a UI (User Interface) 250. The sensor provided by the portable terminal 200 can be any sensor. The portable terminal 200 is a communication terminal carried by an operator, and as a specific example, it is a smart phone or a tablet terminal. As a specific example, the operator is a person or a robot working in a machine. The operator is equivalent to a data acquirer. As a specific example, the operator's work is the inspection of the equipment that constitutes the machine.

[0047] The air pressure sensor 210 is a sensor that measures air pressure.

[0048] The acceleration sensor 220 is a sensor that measures acceleration of the portable terminal 200 in various directions.

[0049] Angular velocity sensor 230 is a sensor that measures the angular velocity of portable terminal 200 .

[0050] The reconstruction data acquisition sensor 240 is a sensor for acquiring reconstruction data, and is a camera or a distance sensor as a specific example.

[0051] The UI 250 is information indicating an image displayed on a screen included in the portable terminal 200 .

[0052] The equipment device 300 is each device included in the equipment, and acquires a control signal 310 and a sensor signal 320 .

[0053] The control signal 310 is a signal for controlling each device included in the equipment device 300 .

[0054] The sensor signal 320 is a signal obtained by each sensor included in the equipment device 300 .

[0055] The data acquisition device 400 acquires data from the portable terminal 200 and the machine equipment 300 respectively, stores the data acquired from the machine equipment 300 in the machine equipment data DB 410, stores the data acquired from the portable terminal 200 and acquired by each sensor (except the reconstruction data acquisition sensor 240) possessed by the portable terminal 200 in the sensor data DB 420, and stores the data acquired from the portable terminal 200 and acquired by the reconstruction data acquisition sensor 240 in the reconstruction data DB 430.

[0056] The reconstruction data DB 190 is a DB storing reconstruction data to which metadata is added. Metadata is data indicating each part and its status. In addition, metadata indicating the status of the part indicated by the reconstruction data and the date and time when the reconstruction data was acquired is added to the reconstruction data.

[0057] Figure 2 A specific example of metadata when the reconstruction data represents each part of an elevator is shown. Figure 2 As shown in FIG. 1 , the metadata indicates each part indicated by the reconstruction data and the state of each part at the date and time when the reconstruction data was acquired. In addition, the reconstruction data may be data indicating a plurality of parts.

[0058] The equipment device data DB 410 is a DB that stores equipment device data.

[0059] The sensor data DB 420 is a DB that stores sensor data.

[0060] The reconstruction data DB 430 is a DB that stores reconstruction data.

[0061] Figure 3The following is a hardware configuration example of the information processing device 100 of this embodiment. The information processing device 100 is composed of a computer. The information processing device 100 may also be composed of a plurality of computers. The hardware configuration example of each device included in the three-dimensional reconstruction system 90 may be the same as the hardware configuration example of the information processing device 100.

[0062] As shown in the figure, the information processing device 100 is a computer having hardware such as a processor 11, a memory 12, an auxiliary storage device 13, an input / output IF (Interface) 14, and a communication device 15. These hardware are connected via a signal line 19 as appropriate.

[0063] The processor 11 is an IC (Integrated Circuit) that performs calculations and controls hardware included in the computer. As a specific example, the processor 11 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit).

[0064] The information processing device 100 may include a plurality of processors instead of the processor 11. The plurality of processors share the role of the processor 11.

[0065] The memory 12 is typically a volatile storage device, and a specific example is a RAM (Random Access Memory). The memory 12 is also called a main storage device or a main memory. The data stored in the memory 12 is saved in the auxiliary storage device 13 as needed.

[0066] The auxiliary storage device 13 is typically a nonvolatile storage device, and specific examples thereof include ROM (Read Only Memory), HDD (Hard Disk Drive), or flash memory. Data stored in the auxiliary storage device 13 is loaded into the memory 12 as needed.

[0067] The memory 12 and the auxiliary storage device 13 may be integrally formed.

[0068] The input / output IF 14 is a port for connecting an input device and an output device. As a specific example, the input / output IF 14 is a USB (Universal Serial Bus) terminal. As a specific example, the input device is a keyboard and a mouse. As a specific example, the output device is a display.

[0069] The communication device 15 is a receiver and a transmitter. As a specific example, the communication device 15 is a communication chip or a NIC (Network Interface Card).

[0070] Each unit of the information processing device 100 may appropriately use the input / output IF 14 and the communication device 15 when communicating with other devices or the like.

[0071] The auxiliary storage device 13 stores a 3D reconstruction program. The 3D reconstruction program is a program that enables a computer to realize the functions of each unit of the information processing device 100. The 3D reconstruction program is loaded into the memory 12 and executed by the processor 11. The functions of each unit of the information processing device 100 are realized by software.

[0072] The data used when executing the three-dimensional reconstruction program and the data obtained by executing the three-dimensional reconstruction program are appropriately stored in the storage device. Each part of the information processing device 100 appropriately uses the storage device. As a specific example, the storage device is composed of at least one of the memory 12, the auxiliary storage device 13, the register in the processor 11, and the cache memory in the processor 11. In addition, data and information sometimes have the same meaning. The storage device can also be independent of the computer.

[0073] The functions of the memory 12 and the auxiliary storage device 13 may also be implemented by other storage devices.

[0074] The three-dimensional reconstruction program may also be recorded in a non-volatile recording medium that can be read by a computer. As a specific example, the non-volatile recording medium is an optical disk or a flash memory. The three-dimensional reconstruction program may also be provided as a program product.

[0075] ***Description of the action***

[0076] The operation process of the 3D reconstruction system 90 is equivalent to the 3D reconstruction method. The 3D reconstruction method is also a general term for the method executed by each device of the 3D reconstruction system 90. In addition, the program that realizes the operation of the 3D reconstruction system 90 is equivalent to the 3D reconstruction program. The 3D reconstruction program is also a general term for the program executed by each device of the 3D reconstruction system 90.

[0077] Figure 4 4 is a flowchart showing an example of the operation of the data acquisition device 400. Figure 4 The operation of the data acquisition device 400 will be described.

[0078] (Step S101)

[0079] When the operator operates the portable terminal 200 , information indicating an instruction to start data acquisition is input to the data acquisition device 400 .

[0080] In addition, the reconstruction data acquisition sensor 240 acquires data indicating the work target site by the worker operating the portable terminal 200. At this time, as a specific example, the worker uses the portable terminal 200 to capture an image of the work target site.

[0081] (Step S102)

[0082] The data acquisition device 400 starts acquiring each of the equipment data, the sensor data, and the reconstruction data.

[0083] (Step S103)

[0084] The data acquisition device 400 stores the acquired data in each DB as appropriate. At this time, the data acquisition device 400 may convert the acquired data as appropriate and store the converted data in each DB.

[0085] (Step S104)

[0086] The data acquisition device 400 determines whether the termination condition for data acquisition is satisfied. If the termination condition is satisfied, the data acquisition device 400 proceeds to step S105. Otherwise, the data acquisition device 400 proceeds to step S102.

[0087] As a specific example, the termination condition is that an instruction to terminate data acquisition is input to data acquisition device 400 by an operator operating portable terminal 200, a designated acquisition time has passed since data acquisition started, or the amount of data acquired by data acquisition device 400 has reached a designated upper limit.

[0088] (Step S105)

[0089] The data acquisition device 400 ends data acquisition.

[0090] Figure 5 2 is a flowchart showing an example of a process in which the information processing device 100 generates metadata. Figure 5 This process will be described.

[0091] (Step S121)

[0092] First, the work estimation unit 110 reads out the equipment data stored in the equipment data DB 410 and the sensor data stored in the sensor data DB 420 .

[0093] Next, the work estimation unit 110 estimates the work of the operator based on the read data in time series, and estimates the parts of the machine corresponding to the work of the operator and the states of the parts based on the read data and the estimated work. At this time, the work estimation unit 110 estimates the states of the parts at each date and time using information indicating the date and time associated with the read data.

[0094] Next, the work estimation unit 110 inputs data indicating the result of estimating the part and the state of the part to the metadata generation unit 120 .

[0095] Figure 6 A specific example of this step is shown. In this example, the work estimation unit 110 obtains data from the machine equipment data DB 410 and the sensor data DB 420, and estimates the work of the operator, the part, and the state of the part using the obtained data and the estimation logic constructed in advance. Specifically, the work estimation unit 110 estimates the "work place" as the 1st floor based on the data indicating the air pressure, estimates the "work content" as "door inspection" based on the door inspection signal indicating "signal on", estimates the "part" as "1st floor door" based on the estimated "work place" and "work content", and estimates the "part state" as "door open" based on the "part" being "1st floor door" and the "door opening and closing signal" indicating "open". In addition, the work estimation unit 110 can also use machine learning methods to estimate each item.

[0096] (Step S122)

[0097] The metadata generation unit 120 appropriately associates the date and time of obtaining the reconstruction data with the result estimated in time series, and adds metadata indicating the "part" and "part state" respectively to the reconstruction data, and stores the reconstruction data obtained by adding the metadata in the reconstruction data DB 190. Here, the "part state" indicates the state of the "part" whose state may change. As a specific example, when the "part" is a movable part, the "part state" indicates the posture of the "part" or the shape of the "part", and when the color of the light of the "part" changes, the "part state" indicates what color the "part" is.

[0098] Figure 7 2 is a flowchart showing an example of a three-dimensional reconstruction process performed by the information processing device 100. Figure 7 This process will be described.

[0099] (Step S141)

[0100] The grouping unit 130 reads reconstruction data to which metadata is added from the reconstruction data DB 190 , and groups the read reconstruction data according to the part and the state of the part. The grouping unit 130 inputs the grouped reconstruction data to the three-dimensional reconstruction unit 140 .

[0101] (Step S142)

[0102] The 3D reconstruction unit 140 performs 3D reconstruction for each part and its state using the reconstruction data classified by the grouping unit 130 , and inputs data indicating the result of the 3D reconstruction, ie, the 3D reconstruction result, to the integration unit 150 .

[0103] (Step S143)

[0104] First, the integration unit 150 generates a frame for attaching a result of three-dimensional reconstruction according to the position using the design data 191 .

[0105] Next, the integration unit 150 integrates the three-dimensional reconstruction result represented by the input data into the generated frame. At this time, the integration unit 150 adjusts the orientation of each part according to the characteristics of each part, and attaches each part represented by the input data to the position corresponding to each part in the frame. As a specific example, the integration unit 150 attaches the landing operation panel represented by the input data to the frame at the same position in the frame as the position where the landing operation panel is actually installed, in such a way that the orientation of the landing operation panel is the same as the orientation of the actual landing operation panel. In addition, the integration unit 150 makes the states corresponding to each part integrated into a certain frame consistent, that is, makes the states corresponding to each two parts integrated into a certain frame not contradict each other.

[0106] The integration unit 150 inputs data indicating an integration result obtained by integrating the three-dimensional reconstruction results into frames to the reconstruction result output unit 160. Alternatively, the integration unit 150 may prepare frames for each state and generate an integration result for each state.

[0107] Figure 8 This is a diagram for explaining this step using a specific example.

[0108] First, the integration unit 150 generates a frame based on the design data 191 corresponding to the device. In this example, the generated frame represents a three-story building.

[0109] Next, the integration unit 150 integrates the three-dimensional reconstruction result into the generated frame. In this example, the integration unit 150 appropriately integrates the result obtained by performing three-dimensional reconstruction using the reconstruction data grouped with metadata indicating the part such as "3rd floor landing button" into the 3rd floor part of the frame. In addition, the integration unit 150 appropriately integrates the result obtained by performing three-dimensional reconstruction using the reconstruction data grouped with metadata indicating the part such as "1st floor landing button" into the 1st floor part of the frame.

[0110] (Step S144)

[0111] The reconstruction result output unit 160 outputs data indicating the integration result.

[0112] In addition, the reconstruction result output unit 160 may also output the integrated result according to each state of the part. As a specific example, the reconstruction result output unit 160 may also separately output the reconstruction result when the elevator landing door is in the open state and the reconstruction result when the landing door is in the closed state. In this case, the three-dimensional reconstruction result output according to each part state may also be switched by using the user interface.

[0113] ***Description of the Effects of Implementation Method 1***

[0114] As described above, according to this embodiment, metadata indicating the part and the state of the part is added to the reconstruction data. Therefore, according to this embodiment, even if there are multiple parts with similar characteristics in a certain machine, the parts can be distinguished, and data indicating different states can be prevented from being taken into the three-dimensional reconstruction result as noise.

[0115] Furthermore, according to the prior art, in a large-scale machine, when three-dimensional reconstruction is performed using a plurality of reconstruction data acquired in mutually non-adjacent regions, it is impossible to determine what the positional relationship of each reconstruction data should be. As a specific example, when reconstruction data representing part A and reconstruction data representing part B are acquired in a divided state, it is impossible to determine what the positional relationship of part A and part B should be during reconstruction. However, according to the present embodiment, since the reconstruction results of each part are integrated into a frame, three-dimensional reconstruction can be performed even when a plurality of reconstruction data acquired in a divided state is used.

[0116] Furthermore, according to the present embodiment, the three-dimensional reconstruction result of the equipment can be presented according to the state of each part.

[0117] ***Other structures***

[0118] <Modification 1>

[0119] Fig. 9 The following is a diagram showing a configuration example of a three-dimensional reconstruction system 90 according to the present embodiment. As shown in the figure, the three-dimensional reconstruction system 90 further includes an inspection plan DB 440 .

[0120] The inspection plan DB 440 is a DB storing inspection plan data. The inspection plan data is data indicating a plan for workers to inspect equipment, and includes information indicating a work plan for equipment, and as a specific example, indicates equipment inspected by workers and the date and time when the workers inspected the equipment.

[0121] The work estimation unit 110 estimates the work of the operator, the part, and the part status respectively using the inspection plan data in addition to the equipment data and the sensor data. At this time, the work estimation unit 110 estimates the work performed by the operator on the equipment during the target period using the information indicating the work plan for the equipment.

[0122] According to the present modification, the work estimating unit 110 estimates each item using the inspection plan data, and therefore the estimation accuracy of the work estimating unit 110 becomes higher.

[0123] <Modification 2>

[0124] Fig.10 A hardware configuration example of the information processing device 100 according to this modification is shown.

[0125] The information processing device 100 has a processing circuit 18 instead of the processor 11 , the processor 11 and the memory 12 , the processor 11 and the auxiliary storage device 13 , or the processor 11 , the memory 12 , and the auxiliary storage device 13 .

[0126] The processing circuit 18 is hardware that realizes at least a part of each unit included in the information processing device 100 .

[0127] The processing circuit 18 may be dedicated hardware, or may be a processor that executes a program stored in the memory 12 .

[0128] When the processing circuit 18 is dedicated hardware, as a specific example, the processing circuit 18 is a single circuit, a complex circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

[0129] The information processing device 10 may include a plurality of processing circuits instead of the processing circuit 18. The plurality of processing circuits share the role of the processing circuit 18.

[0130] In the information processing device 100 , part of the functions may be implemented by dedicated hardware, and the remaining functions may be implemented by software or firmware.

[0131] As a specific example, the processing circuit 18 is implemented by hardware, software, firmware, or a combination thereof.

[0132] The processor 11, the memory 12, the auxiliary storage device 13, and the processing circuit 18 are collectively referred to as a “processing circuit.” That is, the functions of each functional component of the information processing device 100 are realized by the processing circuit.

[0133] The other devices may have the same structure as that of this modification.

[0134] Implementation method 2.

[0135] Hereinafter, mainly points different from the above-described embodiment will be described with reference to the drawings.

[0136] ***Description of the structure***

[0137] Fig.11 FIG. 8 shows a configuration example of a three-dimensional reconstruction system 90 according to this embodiment. Fig.11 As shown, the information processing device 100 further includes a feature analysis unit 170 .

[0138] The feature analysis unit 170 receives reconstruction data from the reconstruction data DB 430 and analyzes features represented by the received reconstruction data. In addition, the feature analysis unit 170 estimates the part represented by the reconstruction data and the state of the part based on the features represented by the reconstruction data. The feature analysis unit 170 may also estimate the part of the device as the operation object and the state of the part of the device as the operation object using an inference model that takes at least one of the estimated operation and the feature of the reconstruction data as input and infers the part and the state of the part corresponding to the input. The feature analysis unit 170 may also generate an inference model.

[0139] The metadata generating unit 120 generates metadata based on the part and the state of the part estimated based on the features represented by the reconstruction data.

[0140] ***Description of the action***

[0141] Fig.12 2 is a flowchart showing an example of a process in which the information processing device 100 generates metadata. Fig.12 This process will be described.

[0142] (Step S321)

[0143] The feature analysis unit 170 extracts features of the reconstruction data and estimates the part and part state represented by the reconstruction data based on the extracted features. At this time, the feature analysis unit 170 may use an inference model generated by machine learning.

[0144] The feature analysis unit 170 inputs data indicating the results of estimating the part and the part state to the metadata generation unit 120 .

[0145] (Step S122)

[0146] The metadata generating unit 120 integrates the estimation result obtained by the work estimating unit 110 and the estimation result obtained by the feature analyzing unit 170 .

[0147] Then, the metadata generating section 120 executes the above-mentioned processing.

[0148] Fig.13 2 is a flowchart showing an example of a process in which the information processing device 100 generates metadata. Fig.13 This process will be described.

[0149] (Step S322)

[0150] The feature analysis unit 170 extracts features of the reconstruction data, and uses the results of the estimation of the part and part status by the work estimation unit 110 and the extracted features as inputs, thereby respectively estimating the part and part status represented by the reconstruction data using a machine learning method.

[0151] The feature analysis unit 170 inputs data indicating the results of respectively estimating the part and the part state to the metadata generation unit 120 .

[0152] ***Description of the Effects of Implementation Method 2***

[0153] As described above, according to the present embodiment, the part and the part status are estimated using the features represented by the reconstruction data, so the accuracy of the part and the part status represented by the metadata is improved.

[0154] ***Other Implementations***

[0155] The above-described embodiments may be freely combined, arbitrary components of the embodiments may be modified, or arbitrary components of the embodiments may be omitted.

[0156] The embodiments are not limited to the contents described in Embodiments 1 and 2, and various changes can be made as necessary. The procedures described using flowcharts and the like can also be changed as appropriate.

[0157] Description of symbols

[0158] 11: processor; 12: memory; 13: auxiliary storage device; 14: input / output IF; 15: communication device; 18: processing circuit; 19: signal line; 90: 3D reconstruction system; 100: information processing device; 110: work estimation unit; 120: metadata generation unit; 130: grouping unit; 140: 3D reconstruction unit; 150: integration unit; 160: reconstruction result output unit; 170: feature analysis unit; 190: reconstruction data DB; 191: setting Measurement data; 192: Reference state definition; 200: Portable terminal; 210: Air pressure sensor; 220: Acceleration sensor; 230: Angular velocity sensor; 240: Reconstruction data acquisition sensor; 250: UI; 300: Machinery and equipment; 310: Control signal; 320: Sensor signal; 400: Data acquisition device; 410: Machinery and equipment data DB; 420: Sensor data DB; 430: Reconstruction data DB; 440: Inspection plan DB.

Claims

1. An information processing device for three-dimensionally reconstructing at least a part of a device based on data obtained by observing at least a part of the device, in, The information processing device has: a work estimation unit that estimates work performed by the operator on the equipment during the target period based on at least one of machine equipment data indicating the state of the equipment during the target period and data acquired during the target period by a sensor provided in a terminal operated by the operator, i.e., sensor data, and estimates a part of the equipment that was the work object of the operator during the target period and the state of the part based on a result of the estimated work; as well as A metadata generating unit generates data representing an estimated part and a state of the estimated part as metadata, and assigns the generated metadata to reconstruction data, which represents the part and is acquired by the terminal during the object period and used when performing three-dimensional reconstruction processing.

2. The information processing device according to claim 1, in, The work estimation unit estimates work to be performed on the equipment by the worker during the target period, using information indicating a work plan for the equipment.

3. The information processing device according to claim 1 or 2, in, The information processing device further includes a feature analysis unit that estimates the part and the state of the part based on the features represented by the reconstruction data. The metadata generation unit generates the metadata based on the part and the state of the part estimated based on the feature represented by the reconstruction data.

4. The information processing device according to claim 3, in, The feature analysis unit estimates the part of the device that is the object of the operation and the state of the part of the device that is the object of the operation using an inference model that takes at least one of the features of the estimated operation and the reconstruction data as input to infer the part corresponding to the input and the state of the part.

5. The information processing device according to any one of claims 1 to 4, in, The metadata generation unit assigns metadata to each of the plurality of reconstruction data. The information processing device further comprises: a grouping unit that groups the plurality of reconstruction data according to the parts and states of the parts respectively corresponding to the plurality of reconstruction data; as well as The integration unit generates data representing a frame corresponding to a device having the device, and attaches a result of performing a three-dimensional reconstruction process using the grouped reconstruction data to the generated data representing the frame.

6. A three-dimensional reconstruction system, in, The 3D reconstruction system has: The information processing device according to any one of claims 1 to 5; and The terminal.

7. An information processing method for three-dimensionally reconstructing at least a portion of a device based on data obtained by observing at least a portion of the device, in, The computer estimates the work performed by the operator on the equipment during the target period based on at least one of machine equipment data indicating the state of the equipment during the target period and data obtained by a sensor possessed by a terminal operated by an operator during the target period, i.e., sensor data, and estimates the part of the equipment that was the work object of the operator during the target period and the state of the part based on the result of the estimated work. The computer generates data representing the estimated part and the state of the estimated part as metadata, and assigns the generated metadata to reconstruction data representing the part, which is acquired by the terminal during the target period and used when performing three-dimensional reconstruction processing.

Images