Information processing device and program

Abstract

The invention relates to an information processing device and a program, and provides a technology capable of more accurately inspecting an object using sensing data. An inspection support device (200) according to one embodiment of the present invention: acquires narrow angle data (D22) and wide angle data (D21) in which the shapes of a relatively narrow range and a relatively wide range of an object to be inspected are respectively represented, and the relationship between the positions and postures of sensors (110, 120) at the time of each acquisition is predetermined in advance; a set (narrow-angle data group (DG12) and wide-angle data group (DG11)) of each of narrow-angle data (D12) and wide-angle data (D11), each of which represents a relatively narrow range and a shape of a relatively wide range of an object to be compared, and in which a relationship between positions and orientations of sensors (110, 120) at the time of acquisition of each of the narrow-angle data (D12) and wide-angle data (DG11) is predetermined in advance, is acquired. On the basis of the wide-angle data (D21) and the wide-angle data group (DG11), the narrow-angle data (D22) and narrow-angle data (D12) indicating the same part of the comparison object as the part of the inspection object corresponding to the narrow-angle data (D22) are associated with each other.

Description

Technical Field

[0001] The present invention relates to an information processing device and the like. Background Art

[0002] For example, there is disclosed a technology for inspecting an object using sensor data such as image data (see Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2016-99633 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, when sensor data representing the shape of a relatively narrow range, such as image data with a relatively narrow field of view, is used, the detailed shape of the range can be reflected in the data, so that the state of the range of the object can be determined in more detail.

[0008] However, in the case of sensor data representing a shape in a relatively narrow range, it may be impossible to determine which range of the entire inspection target range of the object the data corresponds to. Therefore, for example, it may be impossible to accurately extract sensor data of the inspection target part from the sensor data group, and as a result, it may be impossible to accurately perform the inspection.

[0009] On the other hand, when sensor data showing a shape in a relatively wide range, such as image data having a relatively wide viewing angle, is used, it is easy to identify a portion corresponding to the data range in the entire inspection target range of the object.

[0010] However, in the sensing data showing the shape of a relatively wide range, the detailed shape is not easily reflected in the data. Therefore, the state of the range included in the sensing data may not be determined in detail, and as a result, the inspection may not be performed accurately.

[0011] Therefore, in view of the above-mentioned problems, an object of the present invention is to provide a technology capable of more accurately inspecting an object using sensor data.

[0012] Means for solving problems

[0013] In order to achieve the above object, in one embodiment of the present invention, an information processing device is provided, which comprises:

[0014] a first acquisition unit that acquires first sensor data indicating a shape of a relatively narrow range of a first object and second sensor data indicating a shape of a relatively wide range of the first object and having a predetermined relationship between positions and postures of sensors when acquiring the first sensor data;

[0015] a second acquisition unit for acquiring a third sensor data group and a fourth sensor data group, each of which is a set of third sensor data representing a shape of a relatively narrow range of a second object that is a comparison target of the first object, and a set of fourth sensor data representing a shape of a relatively wide range of the second object and in which the relationship between the positions and postures of the sensors at the time of acquisition is predetermined with respect to the third sensor data; and

[0016] The correspondence establishing unit establishes a correspondence between the first sensor data and the third sensor data indicating the same portion of the second object as the portion of the first object corresponding to the first sensor data, based on the second sensor data and the fourth sensor data group.

[0017] Furthermore, in another embodiment of the present invention, there is provided an information processing device comprising:

[0018] a first acquisition unit for acquiring first sensor data representing a shape of a relatively narrow range of a first object and second sensor data representing a shape of a relatively wide range of the first object and wherein the relationship between the positions and postures of sensors at the time of acquisition is predetermined with respect to the first sensor data, i.e., a first sensor data group and a second sensor data group;

[0019] a storage unit storing a third sensing data set which is a set of third sensing data representing the shape of the second object and having a predetermined correspondence relationship with a portion of the second object or the three-dimensional shape data; and

[0020] The correspondence establishing unit establishes a correspondence between the first sensing data and the third sensing data indicating the same part of the first object and the second object based on the second sensing data group and the data of the three-dimensional shape of the second object.

[0021] Furthermore, in another embodiment of the present invention, there is provided an information processing device comprising:

[0022] a first acquisition unit that acquires first sensor data representing a relatively wide range of a shape of a first object and second sensor data representing a relatively wide range of the shape of the first object and wherein the relationship between the positions and postures of sensors at the time of acquisition is predetermined with respect to the first sensor data, i.e., a first sensor data group and a second sensor data group; and

[0023] The correspondence establishing unit establishes a correspondence between the second sensing data and a part of the first object indicated by the second sensing data, based on the first sensing data group.

[0024] Furthermore, in another embodiment of the present invention, a program is provided, which causes an information processing device to execute the following steps:

[0025] A first acquisition step of acquiring first sensor data indicating a shape of a relatively narrow range of a first object and second sensor data indicating a shape of a relatively wide range of the first object and wherein the relationship between the positions and postures of sensors at the time of acquisition is predetermined with respect to the first sensor data;

[0026] a second acquisition step of acquiring a third sensor data group and a fourth sensor data group, each of which is a set of third sensor data representing a shape of a relatively narrow range of a second object that is a comparison object of the first object, and a set of fourth sensor data representing a shape of a relatively wide range of the second object and in which the relationship between the positions and postures of the sensors at the time of acquisition is predetermined with respect to the third sensor data; and

[0027] The corresponding relationship establishing step establishes a corresponding relationship between the first sensing data and the third sensing data representing the same portion of the second object as the portion of the first object corresponding to the first sensing data, based on the second sensing data and the fourth sensing data group.

[0028] Effects of the Invention

[0029] According to the above-described embodiment, it is possible to more accurately inspect an object using sensing data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a diagram schematically showing an example of an inspection support system.

[0031] Figure 2 This is a diagram schematically showing another example of the inspection support system.

[0032] Figure 3 It is a figure which shows a specific example of the inspection object.

[0033] Figure 4 It is a figure which shows a specific example of the inspection object.

[0034] Figure 5 It is a figure which shows a specific example of the inspection object.

[0035] Figure 6 This is a diagram showing an example of the hardware configuration of the inspection support device.

[0036] Figure 7 This is a functional block diagram showing a first example of the configuration of the inspection support device.

[0037] Figure 8 This is a functional block diagram showing a second example of the structure of the inspection support device.

[0038] Fig. 9 This is a diagram showing a first example of a screen displayed on a display device.

[0039] Fig.10 This is a diagram showing a second example of a screen displayed on the display device.

[0040] Fig.11 It is a diagram showing a third example of a screen displayed on the display device. DETAILED DESCRIPTION

[0041] Hereinafter, embodiments will be described with reference to the drawings.

[0042] [Overview of the inspection support system]

[0043] First, refer to Figure 1 to Figure 5 An example of the inspection support system SYS according to the present embodiment will be described.

[0044] Figure 1 It is a diagram showing an example of the inspection support system SYS. Figure 2 It is a diagram showing another example of the inspection support system SYS. Figure 3 to Figure 5 : is a diagram showing a specific example of an inspection object. Specifically, Figure 3 It is a side view of a shovel as an example of the inspection object. Figure 4 It is a side view of a mobile crane (crawler crane) which is another example of the inspection object. Figure 5 This is a side view of a continuous unloader as another example of the inspection object.

[0045] The inspection support system SYS supports the user in inspecting the difference between the inspection object and the comparison object (hereinafter referred to as "comparative inspection" for convenience).

[0046] The inspection object is, for example, a large structure. The large structure as the inspection object includes, for example, a large machine. Figure 3 to Figure 5As shown, large machinery includes construction machinery such as excavators, mobile cranes (crawler cranes), and continuous unloaders. In addition, large machinery may also include large ships. In addition, large structures that are inspection objects include, for example, large factories (factory buildings, factory equipment), etc. Large factories include, for example, buildings and equipment of power plants, buildings and equipment of iron and steel plants, etc. In addition, objects to be inspected include large civil structures and building structures. Large civil structures include, for example, road facilities, railway facilities, dams, bridges, airport facilities, port facilities, etc. Large building structures include, for example, super high-rise buildings, large sports facilities (stadiums), etc.

[0047] The comparative inspection includes, for example, an inspection of the change of the inspection object relative to the past state, that is, an inspection of the time-series change of the inspection object. In this case, the comparison object is the inspection object itself in the past. The time-series change of the inspection object includes, for example, the generation of rust, deformation, cracks, loose screws, missing parts, modification, discoloration, relative displacement relative to the surrounding caused by earthquakes, etc. In addition, the comparative inspection includes the inspection of the difference between the inspection object and the comparison object of the same type (same design) as the inspection object. In this case, the comparison object is, for example, a trial product of a product of the same design as the inspection object, and a product that is qualified according to the median tolerance inspection in its initial batch. The difference between different comparison objects of the same type (same design) as the inspection object, for example, includes differences in shape, color, and the presence or absence of parts that exceed the assumed manufacturing error.

[0048] Furthermore, the scope of the comparative inspection may be the entire inspection object and the comparative inspection object, or may be a part thereof. The following description will focus on the case where the scope of the comparative inspection is the entire inspection object and the comparative inspection object.

[0049] Furthermore, there may be a plurality of comparison objects. For example, in the case of an inspection of a time-series change of an inspection object, an inspection is performed by comparing a current inspection object with a plurality of comparison objects corresponding to inspection objects at a plurality of different past times.

[0050] like Figure 1 As shown in FIG. 1 , the inspection support system SYS includes, for example, a sensor device 100 and an inspection support apparatus 200. Figure 2 As shown, the inspection support system SYS may further include a terminal device 300 .

[0051] The sensor device 100 acquires sensor data related to the shape of the inspection object and the comparison object. For example, the sensor device 100 acquires sensor data related to the shape of the inspection object and the comparison object according to the operation of the staff. In addition, the sensor device 100 can acquire sensor data related to the shape of the inspection object and the comparison object according to the movement of the staff holding the sensor device 100. In addition, the sensor device 100 can be configured to be movable, such as a drone. In this case, the sensor device 100 acquires sensor data related to the shape of the inspection object and the comparison object according to the operation from the outside, for example, while moving according to the operation from the outside. In addition, the sensor device 100 can also automatically acquire sensor data related to the shape of the inspection object and the comparison object while moving autonomously.

[0052] The sensor device 100 includes sensors 110 , 120 .

[0053] The sensor device 100 may be a device dedicated to the function of acquiring sensor data, or may be a general-purpose device. For example, the sensor device 100 may be an information device (smart device) equipped with a camera device, a distance sensor, etc. as sensors 110 and 120. Information devices include, for example, smart phones and tablet terminals. Furthermore, the sensor device 100 may also be a drone equipped with a camera device, a distance sensor, etc. as sensors 110 and 120.

[0054] The sensors 110 and 120 acquire sensing data regarding the shapes of the inspection object and the comparison object, respectively.

[0055] For example, sensors 110 and 120 are imaging devices that can obtain a camera image (image data) that captures the shape of the inspection object. The imaging device is, for example, a monocular camera. In addition, the imaging device may be a camera (hereinafter referred to as a "3D camera" for convenience) that can obtain data related to distance (depth) in addition to two-dimensional images, such as a stereo camera, an RGB-D camera, a TOF (Time Of Flight) camera, etc. In addition, sensors 110 and 120 may also be distance sensors. The distance sensor may be, for example, a distance sensor that can obtain point group data corresponding to the shape of the inspection object based on the sensor device 100 (sensors 110 and 120). Distance sensors include, for example, LIDAR (Light Detecting and Ranging), millimeter wave radar, ultrasonic sensors, etc. In addition, sensors 110 and 120 may be one of which is an imaging device and the other is a distance sensor.

[0056] The sensor 110 has a wider sensing range than the sensor 120. Therefore, the sensor 110 can acquire sensing data representing the shape of the inspection object in a wider range than the sensor 120. Hereinafter, for convenience, the sensing data acquired by the sensor 110 may be referred to as "wide-angle data". The sensor 110 is, for example, an omnidirectional camera.

[0057] The sensor 120 has a narrower sensing range than the sensor 110. Thus, the sensor 120 can acquire sensing data representing a detailed shape of a relatively narrow range of the inspection object. Hereinafter, for convenience, the sensing data acquired by the sensor 120 may be referred to as "narrow angle data".

[0058] The relative positions and postures of the sensors 110 and 120 are fixed. Furthermore, the acquisition times of the sensing data of the sensors 110 and 120 are synchronized. Thus, when the sensor device 100 is located at a certain position, the sensors 110 and 120 acquire both wide-angle data and narrow-angle data.

[0059] The sensor data (wide-angle data and narrow-angle data) related to the shapes of the inspection object and the comparison object may be acquired by the same sensor device 100 or by different sensor devices 100. In the latter case, the relative positional relationship between the sensors 110 and 120 may be different in the sensor device 100 that acquires the sensor data related to the shape of the inspection object and the sensor device 100 that acquires the sensor data related to the shape of the comparison object.

[0060] The sensing data (wide-angle data and narrow-angle data) acquired by the sensor device 100 is input to the inspection support apparatus 200 .

[0061] For example, the sensor device 100 sends the sensing data to the inspection support device 200 through a prescribed communication network. The prescribed communication network includes, for example, a wide area network (WAN). The wide area network includes, for example, a mobile communication network with a base station as a terminal, a satellite communication network using a communication satellite, and the Internet. In addition, the prescribed communication network may also include a local area network (LAN). In addition, the prescribed communication network may also include a short-distance communication line based on a communication standard such as Bluetooth (registered trademark) or WiFi. For example, Figure 1 As shown, the sensor device 100 directly transmits the sensing data to the inspection support device 200. Figure 2 As shown, the sensor device 100 may transmit the sensing data to the inspection support device 200 via the terminal device 300 as a relay device.

[0062] Furthermore, the sensing data acquired by the sensor device 100 can be stored in a portable storage medium from the sensor device 100, and input from the storage medium to the inspection support device 200. The storage medium is, for example, a flash memory such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), a USB memory, or an SD card.

[0063] The inspection support device 200 supports the comparative inspection performed by the user. For example, the inspection support device 200 provides the user with information for the comparative inspection based on the sensing data (wide-angle data and narrow-angle data) input from the sensor device 100 .

[0064] The inspection support device 200 is, for example, a terminal device (user terminal) used by a user. The user terminal may be, for example, a fixed terminal device such as a desktop PC (Personal Computer). In addition, the user terminal may also be a portable (mobile) terminal device (portable terminal) such as a smart phone, a tablet terminal, or a notebook PC.

[0065] [Check the hardware structure of the support device]

[0066] Next, refer to Figure 6 The hardware configuration of the inspection support device 200 will be described.

[0067] Figure 6 It is a block diagram showing an example of the hardware configuration of the inspection support apparatus 200 .

[0068] The functions of the inspection support device 200 are implemented by any hardware or any combination of hardware and software. Figure 6 As shown, the inspection support device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU (Central Processing Unit) 204, a high-speed computing device 205, a communication interface 206, an input device 207, a display device 208, and a sound output device 209 connected by a bus BS2.

[0069] The external interface 201 functions as an interface for reading data from the storage medium 201A and writing data to the storage medium 201A. The storage medium 201A includes, for example, a floppy disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, and a USB memory. Thus, the inspection support device 200 can read various data used in the processing through the storage medium 201A, store it in the auxiliary storage device 202, or install programs that realize various functions.

[0070] Furthermore, the inspection support device 200 may acquire various data and programs used in the processing from an external device via the communication interface 206 .

[0071] The auxiliary storage device 202 stores various installed programs, and stores files, data, etc. required for various processes. The auxiliary storage device 202 includes, for example, a HDD, an SSD, and the like.

[0072] When there is an instruction to start the program, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0073] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203 , and realizes various functions related to the inspection support device 200 according to the programs.

[0074] The high-speed computing device 205 performs computing at a relatively high speed in conjunction with the CPU 204. The high-speed computing device 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or a FPGA (Field-Programmable Gate Array).

[0075] In addition, the high-speed computing device 205 may be omitted depending on the required computing processing speed.

[0076] The communication interface 206 is used as an interface for connecting to external devices so as to be communicable. Thus, the inspection support device 200 can communicate with external devices of the inspection support device 200 through the communication interface 206. The communication interface 206 may have a plurality of communication interfaces depending on the communication method with the connected device.

[0077] The input device 207 receives various inputs from the user.

[0078] The input device 207 includes, for example, an input device that receives mechanical operation input from the user (hereinafter referred to as an "operation input device"). The operation device for remote operation may be the operation input device. The operation input device includes, for example, a button, a toggle switch, a lever, a keyboard, a mouse, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, and the like.

[0079] Furthermore, the input device 207 may include a voice input device capable of receiving voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0080] Furthermore, the input device 207 may include a gesture input device capable of receiving gesture input from the user. The gesture input device may include, for example, a camera capable of capturing the gesture state of the user.

[0081] Furthermore, the input device 207 may include a biometric input device capable of receiving biometric input from the user. The biometric input device may include, for example, a camera capable of acquiring image data containing information related to the user's fingerprint or iris.

[0082] The display device 208 displays an information screen and an operation screen for the user of the inspection support device 200. The display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

[0083] The sound output device 209 transmits various information by sound to the user of the inspection support device 200. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[0084] [Functional structure of inspection support device]

[0085] refer to Figure 7 to Figure 11 The functional configuration of the inspection support device 200 will be described.

[0086] Figure 7 2 is a functional block diagram showing an example of the configuration of the inspection support device 200 . Figure 8 2 is a functional block diagram showing another example of the configuration of the inspection support device 200 . Fig. 9This is a diagram showing a first example (screen 900 ) of a screen displayed on the display device 208 . Fig.10 It is a diagram showing a second example (screen 1000 ) of the screen displayed on the display device 208 . Fig.11 It is a diagram showing a third example (screen 1100 ) of the screen displayed on the display device 208 .

[0087] In addition, Figures 9 to 11 2 shows an example of a case where a shovel as an inspection object is inspected for changes relative to a past state.

[0088] The following description will focus on the case where the sensing data is image data.

[0089] like Figure 7 and Figure 8 As shown, the inspection support device 200 includes a data acquisition unit 2001 , a comparison target data generation unit 2002 , and a comparison inspection support unit 2003 .

[0090] The data acquisition unit 2001 acquires sensing data input from the sensor device 100 .

[0091] The comparison object data generation unit 2002 generates a data set (reference data set DS17) representing the overall shape of the comparison object based on the wide-angle data set DG11 and the narrow-angle data set DG12 representing the shape of the comparison object acquired by the data acquisition unit 2001. The wide-angle data set DG11 is a collection of a plurality of wide-angle data D11, and the narrow-angle data set DG12 is a collection of a plurality of narrow-angle data D12.

[0092] For example, each wide-angle data D11 is provided with inherent identification information such as ID (Identification). Similarly, each narrow-angle data D12 is provided with inherent identification information such as ID. As described above, since the sensors 110 and 120 synchronize the acquisition time of the wide-angle data and the narrow-angle data, the identification information of the wide-angle data D11 and the narrow-angle data D12 acquired at the same time is associated with each other through a database or the like. As a result, the inspection support device 200 can extract a combination of wide-angle data D11 and narrow-angle data D12 acquired at the same time (i.e., the time when the sensor device 100 is in the same position and the same posture) from the wide-angle data group DG11 and the narrow-angle data group DG12.

[0093] The comparison target data generating unit 2002 includes an SfM (Structure from Motion) processing unit 2002A, a storage unit 2002B, a narrow-angle data position estimating unit 2002C, a data set generating unit 2002D, and a storage unit 2002E.

[0094] The functions of the SfM processing unit 2002A, the narrow-angle data position estimation unit 2002C, and the data set generation unit 2002D are realized by, for example, loading a program installed in the auxiliary storage device 202 into the memory device 203 and executing it by the CPU 204. In addition, the functions of the storage units 2002B and 2002E are realized by a predetermined storage area in the auxiliary storage device 202 or the like.

[0095] The SfM processing unit 2002A performs a known SfM process based on the wide-angle data group DG11 of the comparison object acquired by the data acquisition unit 2001. As a result, the SfM processing unit 2002A can output a three-dimensional model (3D model) D13 of the comparison object and data (wide-angle position information data D14) of information on the position and posture of the sensor 110 corresponding to the wide-angle data D11 for each wide-angle data D11. Hereinafter, the set of wide-angle position information data D14 for each wide-angle data D11 is referred to as a "wide-angle position information data group DG14".

[0096] The position and posture of the sensor 110 corresponding to the wide-angle data D11 are the relative position and posture of the sensor 110 with respect to the comparison object when the wide-angle data D11 is acquired.

[0097] The sensor relative position information data D15 is pre-stored in the storage unit 2002B.

[0098] The sensor relative position information data D15 is data indicating information on the relationship between the relative positions and postures of the sensors 110 and 120 in the sensor device 100 that has acquired sensing data related to the shape of the comparison object.

[0099] The narrow-angle data position estimation unit 2002C estimates the position and posture of the sensor 120 corresponding to the narrow-angle data D12 for each narrow-angle data D12. Furthermore, the narrow-angle data position estimation unit 2002C outputs data (narrow-angle position information data D16) of information on the position and posture of the sensor 120 corresponding to the narrow-angle data D12 for each narrow-angle data D12. Hereinafter, a set of the narrow-angle position information data D16 for each narrow-angle data D12 may be referred to as a "narrow-angle position information data group DG16".

[0100] The position and posture of the sensor 120 corresponding to the narrow-angle data D12 are the relative position and posture of the sensor 120 with respect to the comparison object when the narrow-angle data D12 is acquired.

[0101] For example, the narrow-angle data position estimation unit 2002C extracts the wide-angle data D11 associated with the identification information from the wide-angle data group DG11 for each narrow-angle data D12. Next, the narrow-angle data position estimation unit 2002C extracts the wide-angle position information data D14 indicating the position and posture of the sensor 110 corresponding to the extracted wide-angle data D11 from the wide-angle position information data group DG14 for each narrow-angle data D12. Then, the narrow-angle data position estimation unit 2002C calculates the position and posture of the sensor 120 corresponding to the narrow-angle data D12 based on the extracted wide-angle position information data D14 and the sensor relative position information data D15 for each narrow-angle data D12.

[0102] The data set generating unit 2002D generates a reference data set DS17 including the wide-angle data set DG11 , the narrow-angle data set DG12 , the 3D model D13 , the wide-angle position information data set DG14 , and the narrow-angle position information data set DG16 .

[0103] The storage unit 2002E stores the reference data set DS17 generated by the data set generating unit 2002D.

[0104] The comparative inspection support unit 2003 supports comparative inspection by the user based on the wide-angle data D21 and narrow-angle data D22 of the inspection object acquired by the data acquisition unit 2001 and the reference data set DS17.

[0105] The wide-angle data D21 and the narrow-angle data D22 are sensing data acquired by the sensors 110 and 120 at the same time, that is, when the sensor device 100 is at the same position and posture.

[0106] like Figure 7 and Figure 8 As shown in FIG. 2 , the comparison inspection support unit 2003 includes a wide-angle data position estimation unit 2003A, a storage unit 2003B, a narrow-angle data position estimation unit 2003C, a comparison object data search unit 2003D, and a display processing unit 2003E. Fig. 9 As shown, the comparison inspection support unit 2003 may include a difference detection unit 2003F.

[0107] The functions of the wide-angle data position estimation unit 2003A, the narrow-angle data position estimation unit 2003C, the comparison target data search unit 2003D, the display processing unit 2003E, and the difference detection unit 2003F are realized by, for example, loading a program installed in the auxiliary storage device 202 into the memory device 203 and executing it by the CPU 204. In addition, the function of the storage unit 2003B is realized by a predetermined storage area in the auxiliary storage device 202 or the like.

[0108] The wide-angle data position estimation unit 2003A estimates the position and posture of the sensor 110 corresponding to the wide-angle data D21 of the inspection object based on the wide-angle data group DG11 and the wide-angle position information data group DG14 of the comparison object included in the reference data set DS17. Then, the wide-angle data position estimation unit 2003A outputs data (wide-angle position information data D23) of information on the position and posture of the sensor 110 corresponding to the wide-angle data D21.

[0109] The position and posture of the sensor 110 corresponding to the wide-angle data D21 of the inspection object are the relative position and posture of the sensor 110 with respect to the inspection object when the wide-angle data D21 is acquired.

[0110] For example, the wide-angle data position estimation unit 2003A uses a known corresponding point search method to extract wide-angle data of one or more inspection objects that have captured the same parts (corresponding points) as the wide-angle data D21 of the inspection object from the wide-angle data group DG11 of the comparison object. Then, the wide-angle data position estimation unit 2003A estimates (calculates) the position and posture of the wide-angle data based on the information of the position and posture of the sensor 110 corresponding to the extracted wide-angle data included in the wide-angle position information data group DG14.

[0111] The sensor relative position information data D24 is pre-stored in the storage unit 2003B.

[0112] The sensor relative position information data D24 is information indicating the relationship between the relative positions and postures of the sensors 110 and 120 in the sensor device 100 that has acquired the sensing data related to the shape of the inspection object.

[0113] The narrow-angle data position estimation unit 2003C estimates the position and posture of the sensor 120 corresponding to the narrow-angle data D22. Then, the narrow-angle data position estimation unit 2003C outputs data (narrow-angle position information data D25) of information on the position and posture of the sensor 120 corresponding to the narrow-angle data D22.

[0114] The position and posture of the sensor 120 corresponding to the narrow-angle data D22 of the inspection object are the relative position and posture of the sensor 120 with respect to the inspection object when the narrow-angle data D22 is acquired.

[0115] For example, the narrow-angle data position estimation unit 2003C calculates the position and posture of the sensor 120 corresponding to the narrow-angle data D22 based on the position information of the sensor 110 corresponding to the wide-angle data D21 included in the wide-angle position information data D23 and the sensor relative position information data D24.

[0116] In addition, when the sensor device 100 that acquires the wide-angle data group DG11 and the narrow-angle data group DG12 of the comparison object is the same as the sensor device 100 that acquires the wide-angle data D21 and the narrow-angle data D22 of the inspection object, the storage unit 2003B can be omitted. In this case, the narrow-angle data position estimation unit 2003C can estimate the position and posture of the sensor 120 corresponding to the narrow-angle data D22 using the sensor relative position information data D15. In addition, the function of the narrow-angle data position estimation unit 2002C is substantially the same as the function of the narrow-angle data position estimation unit 2003C, and is realized by, for example, the same program installed in the auxiliary storage device 202.

[0117] The comparison object data search unit 2003D searches the narrow-angle data group DG12 of the comparison object for narrow-angle data D12 suitable as a comparison object for the narrow-angle data D22 of the inspection object, based on the narrow-angle position information data D25 and the narrow-angle position information data group DG16. The narrow-angle data D12 suitable as a comparison object for the narrow-angle data D22 of the inspection object is the narrow-angle data group DG12 of the comparison object that contains data showing the shape of the same part (common part) as the narrow-angle data D22 of the inspection object (that is, the same part is photographed). As the search result, the comparison object data search unit 2003D outputs one or more narrow-angle data D12 as comparison object narrow-angle data D26.

[0118] The display processing unit 2003E displays the narrow-angle data D22 and the comparison target narrow-angle data D26 in a comparable manner on the display device 208. In addition, the display processing unit 2003E may display an image representing the overall shape of the comparison target object and the inspection target range of the inspection target object based on the 3D model D13 of the comparison target object while displaying the narrow-angle data D22 and the comparison target narrow-angle data D26.

[0119] For example, Fig. 9 As shown, the display processing unit 2003E displays the screen 900 on the display device 208 .

[0120] In screen 900, image 901 corresponding to comparison target narrow angle data D26 and image 902 corresponding to narrow angle data D22 are displayed side by side. Thus, the user can inspect changes of the inspection target and comparison target (shovel) from past states by comparing images 901 and 902 in which the same portion (common portion) of the inspection target and comparison target is captured.

[0121] And, if Fig.10 As shown, the display processing unit 2003E may also display the screen 1000 on the display device 208 .

[0122] In the screen 1000, an image group 1001 corresponding to the narrow angle data group DG12 of the comparison object, an image 1002 corresponding to the narrow angle data D22 of the inspection object, and an image 1003 schematically showing the overall shape of the inspection target range of the comparison object and the inspection object (shovel) are displayed. In addition, in the screen 1000, an image of a 3D model D13 showing the overall shape of the inspection target range of the inspection object and the comparison object may be displayed instead of the image 1003.

[0123] The image group 1001 includes images 1001 - 1 to 1001 - 5 corresponding to the narrow-angle data D12 included in the narrow-angle data group DG12 .

[0124] Image 1001-5 among images 1001-1 to 1001-5 corresponds to comparison target narrow-angle data D26 and is emphasized by a thick frame line. Thus, the user can easily recognize that image 1001-5 in comparison target object image group 1001 captures the same portion (common portion) as image 1002 of the inspection target object.

[0125] Images 1001-1 to 1001-5 and parts P1 to P5 of image 1003 are connected by a curve and are in correspondence with each other. Also, similar to image 1001-5, image 1002 and part P5 of image 1003 are connected by a curve and are in correspondence with each other. Thus, the user can easily understand which part of the inspection object (excavator) images 1001-1 to 1001-5 and 1200 are images of.

[0126] For example, the comparison inspection support unit 2003 specifies the part of the comparison object captured in the narrow-angle data D12 on the 3D model D13 for each narrow-angle data D12 included in the narrow-angle data group DG12 based on the narrow-angle position information data group DG16. Thus, the display processing unit 2003E can establish a correspondence relationship between the images 1001-1 to 1001-5 corresponding to the narrow-angle data D12 included in the narrow-angle data group DG12 and the part of the image 1003 representing the overall shape of the inspection target range of the comparison object.

[0127] And, if Fig.11 As shown, the display processing unit 2003E may also display the screen 1100 on the display device 208 .

[0128] In the screen 1100, Fig. 9 In the same manner as in the case of FIG. 1 , an image 1101 corresponding to the comparison target narrow-angle data D26 and an image 1102 corresponding to the narrow-angle data D22 are displayed side by side.

[0129] In this case, with Fig. 9In a different situation, either the comparison object narrow-angle data D26 or the narrow-angle data D22 is corrected so that the difference in observation effect of the same part (common part) of the comparison object and the inspection object in the image 1101 and the image 1102 is reduced.

[0130] In this example, the image 1101 is generated by correcting the scale and direction of the image corresponding to the comparison target narrow-angle data D26, and is displayed on the screen 1100 in a manner comparable with the image 1102 corresponding to the narrow-angle data D22. Thus, the user can more efficiently inspect the change of the shovel from the past state by comparing the images 1101 and 1102 in which the same part (common part) of the inspection target object and the comparison target object is captured and the difference in the observation effect of the part is relatively small.

[0131] Furthermore, the viewpoint conversion processing of the first image may be performed so that the positions of the first image corresponding to any one of the comparison target narrow-angle data D26 and the narrow-angle data D22 and the sensor 120 corresponding to the second image corresponding to any other one of them coincide with each other. Thus, the display processing unit 2003E can display images (image after viewpoint conversion of the first image and the second image) in which the observation effect of the same part (common part) between the inspection object and the comparison object is the same on the display device 208. Therefore, by comparing two images in which the same part (common part) of the inspection object and the comparison object is photographed and the observation effect of the part is the same, the user can more efficiently perform comparative inspection of the inspection object and the comparison object.

[0132] Return to Figure 8 The difference detection unit 2003F detects the difference in shape of the same portion (common portion) of the comparison object and the inspection object reflected in each of the comparison object narrow-angle data D26 and the narrow-angle data D22.

[0133] For example, the difference detection unit 2003F detects the difference of a predetermined type between the shapes of the same part (common part) of the comparison object and the inspection object corresponding to the comparison object narrow-angle data D26 and the narrow-angle data D22, respectively, by using a recognizer based on known image processing technology, machine learning, etc. The predetermined type of difference includes, for example, damage, dents, a predetermined component mounting posture, the presence or absence of a predetermined component, etc.

[0134] Furthermore, the narrow angle data D26 and the narrow angle data D22 to be compared may be labeled in advance with features related to the shape difference. The labels indicating features related to the shape difference include, for example, a label indicating the presence of damage, a label indicating the presence of dents, a label indicating the lack of parts, and the like.

[0135] Specifically, the difference detection unit 2003F applies, for example, a recognizer based on known image processing technology, machine learning, etc., and recognizes whether there is a feature corresponding to the label of the object in the narrow-angle data D12 of the object for each narrow-angle data D12 and for each type of label. Then, when there is a feature corresponding to the label of the object in the narrow-angle data D12 of the object, the difference detection unit 2003F assigns the label of the object as metadata to the narrow-angle data D12 of the object. And similarly, the difference detection unit 2003F recognizes whether there is a feature corresponding to the label of the object in the narrow-angle data D22 for each type of label. Then, when there is a feature corresponding to the label of the object in the narrow-angle data D22, the difference detection unit 2003F assigns the label of the object as metadata to the narrow-angle data D22. Thus, the difference detection unit 2003F can detect the difference in shape of the same part (common part) of the comparison object and the inspection object based on the difference in labels respectively assigned to the comparison object narrow-angle data D26 and the inspection object narrow-angle data D22.

[0136] Furthermore, the difference detection unit 2003F may also identify whether there is a difference between the comparison object narrow angle data D26 and the narrow angle data D22 for each of a plurality of predetermined differences, thereby determining the content of the shape difference between the same portion (common portion) of the comparison object and the inspection object.

[0137] Furthermore, the difference detection unit 2003F may also apply, for example, a recognizer based on known image processing technology, machine learning, etc., to infer the degree of shape difference between the same part (common part) of the comparison object and the inspection object corresponding to the comparison object narrow angle data D26 and the narrow angle data D22, respectively.

[0138] Furthermore, when there are multiple narrow-angle data D22, the difference detecting unit 2003F can select multiple combinations of the comparison target narrow-angle data D26 and the narrow-angle data D22 according to the content and degree of the shape difference between the same part (common part) of the comparison target object and the inspection target object.

[0139] When the display processing unit 2003E displays the comparison target narrow-angle data D26 and the narrow-angle data D22 on the display device 208 , the detection result of the difference detection unit 2003F may be reflected in the display content.

[0140] For example, the display processing unit 2003E displays the comparison target narrow-angle data D26 and the narrow-angle data D22 on the display device 208 in a manner that emphasizes the difference in shape between the comparison target object and the inspection target object corresponding to the comparison target narrow-angle data D26 and the narrow-angle data D22, respectively. Specifically, the display processing unit 2003E may display a mark on the image portion where a difference occurs between the two images corresponding to the comparison target narrow-angle data D26 and the narrow-angle data D22. Furthermore, when the shape of the inspection target object corresponding to the narrow-angle data D22 has a feature that is not present in the shape of the comparison target object corresponding to the comparison target narrow-angle data D26, the display processing unit 2003E may display the narrow-angle data D22 on the display device 208 in a manner that makes the feature portion prominent. For example, the display processing unit 2003E performs image processing that exaggerates the amount of damage, dents, etc. of the inspection target object existing in the image corresponding to the narrow-angle data D22, and displays the narrow-angle data D22 on the display device 208.

[0141] Furthermore, the display processing unit 2003E may establish a corresponding relationship between the image representing the overall shape of the inspection target range of the inspection target object and the comparison target object and the information related to the difference of the portion where the difference detection unit 2003F detects a difference in shape between the comparison target object and the inspection target object, and display the same on the display device 208. For example, the display processing unit 2003E establishes a corresponding relationship between the information related to the difference and the portion of the image representing the overall shape of the inspection target range of the inspection target object and the comparison target object corresponding to the portion of the comparison target object and the inspection target object where the difference in shape is detected by the difference detection unit 2003F, and displays the same. The information related to the difference includes, for example, summary information indicating the content and degree of the difference.

[0142] Furthermore, when there are a plurality of narrow-angle data D22, the display processing unit 2003E may distinguish and display a plurality of combinations of the comparison target narrow-angle data D26 and the narrow-angle data D22 on the display device 208 according to the content and degree of the difference selected by the difference detection unit 2003F.

[0143] In this way, the inspection support device 200 can estimate the position and posture of the sensor 110 corresponding to the wide-angle data D11 based on the wide-angle data group DG11 of the comparison object. Therefore, the inspection support device 200 can estimate the position and posture of the sensor 120 corresponding to the narrow-angle data D12 based on the sensor relative position information data D15, under the premise that the relative positional relationship between the sensors 110 and 120 is fixed. In addition, the inspection support device 200 can estimate the position and posture of the sensor 110 corresponding to the wide-angle data D21 of the inspection object based on the wide-angle data group DG11 of the comparison object and the information on the position and posture of the sensor 110 corresponding to the wide-angle data D11. Therefore, the inspection support device 200 can estimate the position and posture of the sensor 120 corresponding to the narrow-angle data D22 of the inspection object based on the sensor relative position information data D24, under the premise that the relative positional relationship between the sensors 110 and 120 is fixed. Therefore, the inspection support device 200 can extract the combination of narrow-angle data D12 and D22 representing the shape of the same part (common part) of the inspection object and the comparison object based on the information of the position and posture of the sensor 120 corresponding to the narrow-angle data D12 and the narrow-angle data D22. Therefore, the user can grasp the overall shape of the inspection range of the comparison object and the inspection object based on the wide-angle data D11 and D21, and grasp the detailed shape of the same part (common part) of the comparison object and the inspection object based on the narrow-angle data D12 and D22. As a result, the user can perform a more accurate comparative inspection.

[0144] [Another embodiment]

[0145] Next, another embodiment will be described.

[0146] The above-described embodiment can be modified and altered as appropriate.

[0147] For example, in the above-mentioned embodiment, a 3D model D13 of the comparison object and a narrow-angle data group DG12 in which the comparison object or a part in the 3D model D13 is specified for each narrow-angle data D12 may be prepared in advance and stored in a predetermined storage unit. The 3D model D13 of the comparison object may be, for example, a three-dimensional CAD (Computer Aided Design) model of the final specifications when designing the comparison object.

[0148] In this case, the wide-angle data group DG11 of the comparison object may be omitted, and the narrow-angle data position estimation unit 2002C may estimate the position and posture of the sensor 120 corresponding to the narrow-angle data D12 based on the 3D model D13 for each narrow-angle data D12. For example, the narrow-angle data position estimation unit 2002C applies a known corresponding point search method based on the narrow-angle data D12 and the data of the shape of the part of the 3D model D13 corresponding to the part of the comparison object determined for the narrow-angle data D12 for each narrow-angle data D12. Thus, the narrow-angle data position estimation unit 2002C can estimate the position and posture of the sensor 120 corresponding to the narrow-angle data D12 on the coordinate system of the 3D model D13 based on the combination of the plurality of corresponding points obtained as the search results.

[0149] Furthermore, at this time, the wide-angle data position estimation unit 2003A can implement the known SfM processing using the wide-angle data group composed of a plurality of wide-angle data D21 of the inspection object. Thus, the wide-angle data position estimation unit 2003A can generate a three-dimensional model of the inspection object and estimate the position and posture of the sensor 110 corresponding to each wide-angle data D21. At this time, the coordinate system showing the three-dimensional model of the inspection object and the position and posture of the sensor 120 corresponding to each wide-angle data D21 is processed to be the same as the coordinate system of the 3D model D13 of the comparison object. For example, based on the comparison between the three-dimensional model of the inspection object obtained by the SfM processing and the 3D model D13 of the comparison object, the coordinate system of the three-dimensional model of the inspection object is made consistent with the coordinate system of the 3D model D13. Thus, the comparison object data search unit 2003D can search for the comparison object narrow-angle data D26 corresponding to the narrow-angle data D22 of the object based on the narrow-angle position information data group DG16 and the narrow-angle position information data D25 shown in the coordinate system of the 3D model D13.

[0150] Furthermore, in the above-mentioned embodiment, the user can perform an inspection on a single body of the inspection object (hereinafter referred to as "single body inspection") without using a comparison object, based on the narrow-angle data D22 of the inspection object, instead of a comparison inspection or as a preliminary inspection before a detailed comparison inspection. This is because, for example, damage, dents, or the presence or absence of missing parts, etc., can sometimes be inspected on a single body of the inspection object.

[0151] In this case, the comparison object data generating unit 2002 is omitted. In addition, the wide-angle data position estimating unit 2003A can implement the known SfM processing using the wide-angle data group composed of a plurality of wide-angle data D21 of the inspection object. Thus, the wide-angle data position estimating unit 2003A can generate a three-dimensional model of the inspection object and estimate the position and posture of the sensor 120 corresponding to each wide-angle data D21. In addition, the comparison object data searching unit 2003D is replaced by the following functional unit: the part of the inspection object corresponding to the narrow-angle data D22 is determined based on the narrow-angle position information data D25, and the narrow-angle data D22 and the part are established in correspondence using a database or the like. Thus, for example, when the display processing unit 2003E displays the narrow-angle data D22 and the image representing the overall shape of the inspection object range of the inspection object on the display device 208, the narrow-angle data D22 and the corresponding part on the image representing the overall shape can be established in correspondence. Therefore, the user can understand the part of the inspection object represented by the narrow-angle data D22. Therefore, the user can grasp the overall shape of the inspection object based on the wide-angle data D21 and grasp the detailed shape of each part of the inspection object based on the narrow-angle data D22. As a result, the user can perform single-body inspection more accurately.

[0152] Furthermore, in the above-mentioned embodiment, its variants, and changed examples, instead of displaying the narrow-angle data D22 and the narrow-angle data D26 to be compared on the display device 208, or in addition thereto, the data of the combination of the narrow-angle data D22 and the narrow-angle data D26 to be compared may be transmitted to the outside. For example, the inspection support device 200 transmits the data of the combination of the narrow-angle data D22 and the narrow-angle data D26 to the terminal device 300. Thus, for example, the user can receive information related to the combination of the narrow-angle data D22 and the narrow-angle data D26 to be compared by using the function of the inspection support device 200 on the server side through the terminal device 300 on the client side. Therefore, the user can perform comparative inspection without being restricted by the place where the inspection support device 200 is installed.

[0153] Furthermore, in the above-mentioned embodiment, its variations and modifications, the sensor device 100 may be built in the inspection support device 200 or the terminal device 300. For example, the inspection support device 200 or the terminal device 300 is a smartphone or a tablet terminal equipped with a camera, LIDAR or the like as the sensor device 100.

[0154] [effect]

[0155] Next, the operation of the information processing device according to this embodiment will be described.

[0156] In this embodiment, the information processing device includes a first acquisition unit, a second acquisition unit, and a correspondence establishment unit. The information processing device is, for example, the inspection support device 200. The first acquisition unit and the second acquisition unit are, for example, the data acquisition unit 2001. The correspondence establishment unit is, for example, the comparison object data search unit 2003D. In addition, the functions of the first acquisition unit, the second acquisition unit, and the correspondence establishment unit can be realized by causing the information processing device to execute a program of a first acquisition step, a second acquisition step, and a correspondence establishment step corresponding to the first acquisition unit, the second acquisition unit, and the correspondence establishment unit, respectively. Specifically, the first acquisition unit acquires the first sensor data representing the shape of a relatively narrow range of the first object, and the second sensor data representing the shape of a relatively wide range of the first object and the relationship between the positions and postures of the sensors when acquiring the first sensor data is predetermined in advance. The first object is, for example, the inspection object. The first sensor data is, for example, the narrow angle data D22. The sensors are, for example, the sensors 110 and 120. The second sensing data is, for example, the wide-angle data D21. Furthermore, the second acquisition unit acquires the third sensing data representing the shape of a relatively narrow range of the second object, which is a comparison object of the first object, and the fourth sensing data representing the shape of a relatively wide range of the second object and whose relationship of the positions and postures of the sensors at the time of acquisition is predetermined with respect to the third sensing data, which are sets of the third sensing data group and the fourth sensing data group. The second object is, for example, the comparison object. The third sensing data is, for example, the narrow-angle data D12. The sensors are, for example, the sensors 110 and 120. The fourth sensing data is, for example, the wide-angle data D11. The third sensing data group is, for example, the narrow-angle data group DG12. The fourth sensing data group is, for example, the wide-angle data group DG11. Moreover, the correspondence establishing unit establishes a correspondence between the first sensor data and the third sensor data based on the second sensor data and the fourth sensor data group, and the third sensor data corresponds to the same part (common part) in the second object as the part of the first object corresponding to the first sensor data (i.e., represents the same part).

[0157] For example, when using sensor data representing the shape of a relatively narrow range, such as image data with a relatively narrow field of view, the detailed shape of the range can be reflected in the data, so that the state of the range of the object can be determined in more detail.

[0158] However, in the case of sensor data representing a shape in a relatively narrow range, it may be impossible to determine which range of the entire inspection target range of the object the data corresponds to. Therefore, for example, it may be impossible to accurately extract sensor data of the inspection target part from the sensor data group, and as a result, it may be impossible to accurately perform the inspection.

[0159] On the other hand, when sensor data showing a shape in a relatively wide range, such as image data having a relatively wide viewing angle, is used, it is easy to identify a portion corresponding to the range of the data in the entire inspection target range of the object.

[0160] However, in the sensing data showing the shape of a relatively wide range, the detailed shape is not easily reflected in the data. Therefore, the state of the range included in the sensing data may not be determined in detail, and as a result, the inspection may not be performed accurately.

[0161] In contrast, the information processing device can, for example, capture the overall shape of the inspection object range of the first object and the second object based on the second sensor data and the fourth sensor data group representing a relatively wide range of shapes. Furthermore, the information processing device can, for example, grasp the part of the first object represented by the first sensor data representing a relatively narrow range of shapes and the part of the second object represented by the third sensor data representing a relatively narrow range of shapes based on the relationship between the positions and postures of the sensors. Therefore, the information processing device can establish a correspondence between the first sensor data and the third sensor data corresponding to the same part (common part) of the first object and the second object. Therefore, the user can use the first sensor data and the third sensor data representing the relatively narrow range of the first object and the second object to perform a more accurate comparative inspection.

[0162] Furthermore, in this embodiment, the first object and the second object may be the same object at different times. Furthermore, the first object and the second object may be different objects of the same design.

[0163] Thus, the information processing apparatus can support inspection of time-series changes of the same object or inspection of differences between different objects of the same design.

[0164] Furthermore, in the present embodiment, the information processing device may include a first inference unit, a second inference unit, a third inference unit, and a fourth inference unit. The first inference unit is, for example, the SfM processing unit 2002A. The second inference unit is, for example, the narrow-angle data position inference unit 2002C. The third inference unit is, for example, the wide-angle data position inference unit 2003A. The fourth inference unit is, for example, the narrow-angle data position inference unit 2003C. Specifically, the first inference unit infers the position and posture of the sensor corresponding to the fourth sensor data for each fourth sensor data based on the fourth sensor data group. Furthermore, the second inference unit infers the position and posture of the sensor corresponding to the third sensor data for each third sensor data based on the inference result of the first inference unit. The third inference unit infers the position and posture of the sensor corresponding to the second sensor data based on the fourth sensor data group and the inference result of the first inference unit. The fourth inference unit infers the position and posture of the sensor corresponding to the first sensor data based on the inference result of the third inference unit. Moreover, the correspondence establishing unit can establish a correspondence between the first sensor data and the third sensor data based on the inference result of the fourth inference unit on the position and posture of the sensor corresponding to the first sensor data and the inference result of the second inference unit on the position and posture of the sensor corresponding to the third sensor data for each third sensor data, and the third sensor data corresponds to a portion (common portion) in the second object that is the same as the portion of the first object corresponding to the first sensor data (i.e., represents the same portion).

[0165] Thus, the information processing device can grasp the position and posture of the sensor corresponding to the first sensor data and the third sensor data, respectively, and can establish a correspondence between the first sensor data and the third sensor data corresponding to the same part (common part) of the first object and the second object.

[0166] Furthermore, in this embodiment, the first inference unit can infer the position and posture of the sensor corresponding to each fourth sensor data based on the relationship between the fourth sensor data representing the same part (common part) of the second object included in the fourth sensor data group.

[0167] Thereby, the information processing device can estimate the position and posture of the sensor corresponding to the fourth sensing data for each fourth sensing data based on the fourth sensing data group.

[0168] Furthermore, in this embodiment, the information processing device may include a detection unit. The detection unit is, for example, a difference detection unit 2003F. Specifically, the detection unit may detect the difference between the same parts (common parts) of the first object and the second object based on the first sensing data and the third sensing data that have been established in a correspondence relationship by the correspondence establishment unit.

[0169] Thus, the user can perform comparative inspection using the detection result. Therefore, the information processing device can improve the convenience of the user and improve the efficiency of comparative inspection.

[0170] Furthermore, in the present embodiment, the detection unit may select the portion where the difference is detected among the same portions (common portions) of the first object and the second object based on at least one of the content of the difference and the degree of the difference.

[0171] Thus, for example, the user can be provided with only the following information: a combination of the first sensing data and the third sensing data corresponding to a portion defined by at least one of the difference content and the difference degree in the same portion (common portion) of the first object and the second object. Therefore, the information processing device can improve the convenience of the user and improve the efficiency of the comparative inspection.

[0172] Furthermore, in this embodiment, the detection unit may also label the first sensor data and the third sensor data that have established a correspondence relationship through the correspondence establishing unit, respectively, to represent characteristics, and detect the difference between the same parts (common parts) of the first object and the second object based on the difference between the labels of each data.

[0173] Thereby, the information processing device can detect the difference between the same portion (common portion) of the first object and the second object.

[0174] Furthermore, in the present embodiment, the information processing device may include a display unit that displays the first sensing data and the third sensing data associated with each other by the association establishing unit.

[0175] Thus, the information processing device can provide the user with information of the first sensing data and the third sensing data corresponding to the same portion (common portion) of the first object and the second object by visual means.

[0176] Furthermore, in the present embodiment, the display unit may emphasize and display the difference in shape between the first sensing data and the third sensing data associated with each other by the association establishing unit.

[0177] Thus, the information processing device can more easily grasp the shape difference between the first sensing data and the third sensing data corresponding to the same part (common part) of the first object and the second object. Therefore, the information processing device can improve the convenience of the user and the efficiency of the comparative inspection.

[0178] Furthermore, in the present embodiment, the display unit may display an image showing the overall shape of the inspection target range of the first object and the second object.

[0179] Thus, the user can perform comparative inspection while confirming not only the combination of sensor data of a specific part but also an image showing the overall shape of the inspection target range of the first object and the second object. Therefore, the information processing device can improve the convenience of the user and improve the efficiency of comparative inspection.

[0180] Furthermore, in the present embodiment, the display unit may associate the first sensing data and the third sensing data associated with the association by the association establishing unit with a part of the entire image and display the associated data.

[0181] Thus, the user can easily grasp the location of the part corresponding to the first sensor data and the third sensor data displayed on the display unit while comparing the part with the image representing the overall shape of the inspection target range of the first object and the second object. Therefore, the information processing device can improve the convenience of the user and improve the efficiency of the comparative inspection.

[0182] Furthermore, in the present embodiment, the display unit can establish a correspondence between (i.e., represented by) the first sensor data and the third sensor data with which a correspondence has been established by the correspondence establishing unit, and information related to the differences between the same parts (common parts) of the first object and the second object, and parts of an image representing the overall shape of the inspection object range of the first object and the second object, and display the same.

[0183] Thus, the user can easily grasp the locations of the parts of the first object and the second object corresponding to the first sensing data and the third sensing data displayed on the display unit while comparing them with the parts of the image representing the overall shapes of the first object and the second object. Therefore, the information processing device can improve the convenience of the user and improve the efficiency of the comparative inspection.

[0184] Furthermore, in this embodiment, the information processing device may include a first acquisition unit, a storage unit, and a correspondence establishing unit. Specifically, the first acquisition unit acquires a first sensor data group and a second sensor data group, which are sets of first sensor data representing a shape of a relatively narrow range of the first object, and second sensor data representing a shape of a relatively wide range of the first object and in which the relationship between the positions and postures of the sensors when acquiring the first sensor data is predetermined in advance. Furthermore, the storage unit stores data of a three-dimensional shape of an inspection target range of a second object as a comparison target of the first object, and a set of third sensor data representing the shape of the second object and in which the correspondence relationship with the part of the second object or the three-dimensional shape data is predetermined in advance. Furthermore, the correspondence establishing unit establishes a correspondence between the first sensor data group and the third sensor data corresponding to the same part (common part) of the first object and the second object (i.e., representing the same part), based on the second sensor data group and the data of the three-dimensional shape of the second object.

[0185] Thus, the information processing device can, for example, capture the overall shape of the inspection object range of the first object and the second object based on the second sensor data group and the data of the three-dimensional shape. Furthermore, the information processing device can, for example, grasp the part of the first object represented by the first sensor data representing a shape in a relatively narrow range and the part of the second object represented by the third sensor data representing a shape in a relatively narrow range based on the relationship between the positions and postures of the sensors. Therefore, the information processing device can establish a correspondence between the first sensor data and the third sensor data corresponding to the same part (common part) of the first object and the second object. Therefore, the user can use the first sensor data and the third sensor data representing the relatively narrow range of the first object and the second object to perform a more accurate comparative inspection.

[0186] Furthermore, in this embodiment, the information processing device may include a first acquisition unit and a correspondence establishing unit. Specifically, the first acquisition unit acquires a first sensor data group and a second sensor data group, which are sets of first sensor data representing a relatively wide range of the shape of the first object and second sensor data representing a relatively wide range of the shape of the first object and in which the relationship between the positions and postures of the sensors at the time of acquisition is predetermined with respect to the first sensor data. Furthermore, the correspondence establishing unit establishes a correspondence between the second sensor data and the part of the first object corresponding to (i.e., represented by) the second sensor data based on the first sensor data group.

[0187] Thus, the information processing device can, for example, capture the overall shape of the inspection target range of the first object based on the second sensor data group. Furthermore, the information processing device can, for example, grasp the part of the first object represented by the first sensor data representing the shape of a relatively narrow range based on the relationship between the positions and postures of the sensors. Therefore, the information processing device can establish a correspondence between the second sensor data and the part of the first object corresponding to the second sensor data. Therefore, the user can use the first sensor data to more accurately perform a single body inspection based on grasping the part of the first object corresponding to the first sensor data representing the relatively narrow range of the first object.

[0188] As mentioned above, although embodiment is described in detail, this invention is not limited to this specific embodiment, Various deformation|transformation and change are possible within the range of the summary described in a claim.

[0189] Finally, this application claims priority based on Japanese patent application No. 2022-156595 filed on September 29, 2022, and the entire contents of the Japanese patent application are incorporated by reference in this application.

[0190] Explanation of symbols

[0191] 100-sensor device, 110-sensor, 120-sensor, 200-inspection support device, 300-terminal device, 2001-data acquisition unit, 2002-comparison object data generation unit, 2002A-SfM processing unit, 2002B-storage unit, 2002C-narrow-angle data position estimation unit, 2002D-data set generation unit, 2002E-storage unit, 2003-comparison inspection support unit, 2003A-wide-angle data position estimation unit, 2003B-storage unit, 2003C-narrow-angle data position estimation unit, 2003D-comparison object data search unit, 2003E-display processing unit, 2003F -Difference detection unit, D11-wide-angle data, D12-narrow-angle data, D13-three-dimensional model, D14-wide-angle position information data, D15-sensor relative position information data, D16-narrow-angle position information data, D21-wide-angle data, D22-narrow-angle data, D23-wide-angle position information data, D24-sensor relative position information data, D25-narrow-angle position information data, D26-comparison object narrow-angle data, DG11-wide-angle data group, DG12-narrow-angle data group, DG14-wide-angle position information data group, DG16-narrow-angle position information data group, DS17-benchmark data set, SYS-inspection support system.

Claims

1. An information processing device comprising: a first acquisition unit that acquires first sensor data indicating a shape of a relatively narrow range of a first object, and second sensor data indicating a shape of a relatively wide range of the first object and having a predetermined relationship between positions and postures of sensors when acquiring the first sensor data; a second acquisition unit for acquiring third sensor data representing a shape of a relatively narrow range of a second object that is a comparison target of the first object, and fourth sensor data representing a shape of a relatively wide range of the second object and wherein the relationship between the positions and postures of sensors at the time of acquisition is predetermined with respect to the third sensor data, i.e., a third sensor data group and a fourth sensor data group, respectively; and The correspondence establishing unit establishes a correspondence between the first sensor data and the third sensor data indicating the same portion of the second object as the portion of the first object corresponding to the first sensor data, based on the second sensor data and the fourth sensor data group.

2. The information processing device according to claim 1, wherein: The first object and the second object are the same object at different times or different objects of the same design.

3. The information processing device according to claim 2, wherein: have: a first inference unit that infers, for each of the fourth sensing data, a position and a posture of a sensor corresponding to the fourth sensing data based on the fourth sensing data group; a second inference unit that infers a position and a posture of a sensor corresponding to the third sensing data for each of the third sensing data based on the inference result of the first inference unit; a third inference unit that infers a position and a posture of a sensor corresponding to the second sensor data based on the fourth sensor data group and the inference result of the first inference unit; and a fourth inference unit that infers the position and posture of the sensor corresponding to the first sensing data based on the inference result of the third inference unit, The correspondence establishing unit establishes a correspondence between the first sensor data and the third sensor data representing the same part of the second object as the part of the first object corresponding to the first sensor data, based on the inference result of the fourth inference unit on the position and posture of the sensor corresponding to the first sensor data and the inference result of the second inference unit on the position and posture of the sensor corresponding to each of the third sensor data.

4. The information processing device according to claim 3, wherein: The first estimation unit estimates the position and posture of a sensor corresponding to each of the fourth sensor data based on a relationship between the fourth sensor data included in the fourth sensor data group and indicating the same portion of the second object.

5. The information processing device according to any one of claims 1 to 4, wherein: A detection unit is provided for detecting a difference between the same parts of the first object and the second object based on the first sensing data and the third sensing data associated with each other by the association establishing unit.

6. The information processing device according to claim 5, wherein: The detection unit selects a portion where a difference is detected among the same portions of the first object and the second object based on at least one of the content of the difference and the degree of the difference.

7. The information processing device according to claim 5, wherein: The detection unit labels the first sensing data and the third sensing data respectively corresponding to each other by the corresponding relationship establishing unit, and detects the difference between the same parts of the first object and the second object based on the difference between the labels of each data.

8. The information processing device according to any one of claims 1 to 4, wherein: A display unit is provided for displaying the first sensing data and the third sensing data associated with each other by the association establishing unit.

9. The information processing device according to claim 8, wherein: The display unit displays a difference in shape between the first sensing data and the third sensing data associated with each other by the association establishing unit with emphasis.

10. The information processing device according to claim 8, wherein: The display unit displays an image showing the overall shape of the inspection target range of the first object and the second object.

11. The information processing device according to claim 10, wherein: The display unit associates the first sensing data and the third sensing data associated with each other by the association establishing unit with a part of the image and displays the associated data.

12. The information processing device according to claim 10, wherein: The display unit establishes a correspondence between the first sensor data and the third sensor data established by the correspondence establishing unit and displays the information related to the difference between the same parts of the first object and the second object, and the correspondence between the parts of the image.

13. An information processing device comprising: a first acquisition unit for acquiring first sensor data representing a shape of a relatively narrow range of a first object, and second sensor data representing a shape of a relatively wide range of the first object and wherein the relationship between the positions and postures of sensors at the time of acquisition is predetermined with respect to the first sensor data, i.e., a first sensor data group and a second sensor data group; a storage unit storing data of a three-dimensional shape of a second object as a comparison object of the first object, and a third sensing data group which is a set of third sensing data representing the shape of the second object and having a predetermined correspondence relationship with a part of the second object or the three-dimensional shape data; and The correspondence establishing unit establishes a correspondence between the first sensing data and the third sensing data indicating the same part of the first object and the second object based on the second sensing data group and the data of the three-dimensional shape of the second object.

14. An information processing device comprising: a first acquisition unit that acquires first sensor data representing a relatively wide range of a shape of a first object, and second sensor data representing a relatively wide range of the shape of the first object and wherein the relationship between the positions and postures of sensors at the time of acquisition is predetermined with respect to the first sensor data, i.e., a first sensor data group and a second sensor data group; and The correspondence establishing unit establishes a correspondence between the second sensing data and a part of the first object indicated by the second sensing data, based on the first sensing data group.

15. A program causing an information processing device to execute the following steps: A first acquisition step of acquiring first sensor data indicating a shape of a relatively narrow range of a first object, and second sensor data indicating a shape of a relatively wide range of the first object and wherein the relationship between the positions and postures of sensors at the time of acquisition is predetermined with respect to the first sensor data; a second acquisition step of acquiring a third sensor data group and a fourth sensor data group, each of which is a set of third sensor data representing a relatively narrow shape of a second object that is a comparison object of the first object, and fourth sensor data representing a relatively wide shape of the second object and wherein the relationship between the positions and postures of the sensors at the time of acquisition is predetermined with respect to the third sensor data; and The corresponding relationship establishing step establishes a corresponding relationship between the first sensing data and the third sensing data representing the same portion of the second object as the portion of the first object corresponding to the first sensing data, based on the second sensing data and the fourth sensing data group.

Images