Information processing device, information processing method, and program

Abstract

Provided is a technology that enables the inspection of differences between large-scale structures to be compared. An inspection assistance apparatus 100 according to one embodiment of the present disclosure is provided with a link processing unit 1104, 1104A, 1104B that, in order to make it possible to compare portions common between one large-scale structure to be compared and an other large-scale structure on the basis of a sensing data group relating to the shape of said one large-scale structure and three-dimensional shape data relating to said one large-scale structure, adjusts the relationship between the sensing data group relating to the shape of said one large-scale structure or the three-dimensional shape data relating to said one large-scale structure and sensing data relating to the shape of the other large-scale structure.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and the like. [Background technology]

[0002] For example, a technology has been disclosed that compares images of the devices being compared taken with a camera to inspect differences between the devices (for example, changes between the same device at different times or differences between different devices with the same design) (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-99633 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the inspection target is a large machine such as a shovel or a large structure such as a steel mill, it is necessary to acquire a large number of captured images focused on the area to be inspected. Furthermore, it is practically impossible to prepare captured images taken from the same distance and angle each time. Therefore, for example, when inspecting changes over time, even if a group of captured images acquired last time and this time is prepared, it is difficult to extract a combination of images corresponding to the area to be inspected. This may reduce the efficiency of the inspection or even make it impossible to perform the inspection.

[0005] In view of the above problems, an object of the present invention is to provide a technology that can inspect differences between large structures to be compared. [Means for solving the problem]

[0006] In order to achieve the above object, in one embodiment of the present disclosure, a first set of sensing data relating to the shape of a first structure; a first sensing data group in which a correspondence relationship between a portion of the first structure corresponding to each piece of data included therein and a portion of the second structure corresponding to second sensing data relating to the shape of a second structure has not been established; and based on the data of the three-dimensional shape of the first structure, The aforementioned The first sensing data group or the three-dimensional shape data of the first structure and the previous data are compared so that common parts of the second structure can be compared. Record number A reorganization unit that organizes the relationship with the sensing data of 2, An information processing device is provided.

[0007] In another embodiment of the present disclosure, an information processing device that receives a first set of sensing data relating to the shape of a first structure; a first sensing data group in which a correspondence relationship between a portion of the first structure corresponding to each piece of data included therein and a portion of the second structure corresponding to second sensing data relating to the shape of a second structure has not been established; and based on the data of the three-dimensional shape of the first structure, The aforementioned The first sensing data group or the three-dimensional shape data of the first structure and the previous data are compared so that common parts of the second structure can be compared. Record number 2. A sorting step for sorting out the relationship with the sensing data; A method for processing information is provided.

[0008] In still another embodiment of the present disclosure, a first set of sensing data relating to the shape of a first structure; a first sensing data group in which a correspondence relationship between a portion of the first structure corresponding to each piece of data included therein and a portion of the second structure corresponding to second sensing data relating to the shape of a second structure has not been established; and based on the data of the three-dimensional shape of the first structure, The aforementioned The first sensing data group or the three-dimensional shape data of the first structure and the previous data are compared so that common parts of the second structure can be compared. Record number 2. causing the information processing device to execute a sorting step of sorting out the relationship with the sensing data; Programs are offered. [Effects of the Invention]

[0009] According to the above-described embodiment, differences between large structures to be compared can be inspected. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of an inspection support system. [Figure 2] FIG. 1 is a diagram showing a specific example of a large structure to be inspected by the inspection support system. [Figure 3] FIG. 1 is a diagram showing a specific example of a large structure to be inspected by the inspection support system. [Figure 4] FIG. 1 is a diagram showing a specific example of a large structure to be inspected by the inspection support system. [Figure 5] 1 is a functional block diagram showing a first example of the configuration of an examination support device. [Figure 6] 10 is a flowchart illustrating an example of a preliminary process for supporting a comparison inspection by a control device. [Figure 7] FIG. 10 is a diagram for schematically explaining the contents of pre-processing related to inspection support by the control device. [Figure 8] FIG. 10 is a diagram for schematically explaining the contents of pre-processing related to inspection support by the control device. [Figure 9] 10 is a flowchart illustrating an example of a main process related to support of a comparison inspection by a control device. [Figure 10] FIG. 10 is a diagram showing an example of a combination of image data of common parts of large structures to be compared, extracted from an image data group. [Figure 11] FIG. 10 is a diagram showing an example of a combination of image data of common parts of large structures to be compared, displayed on a display device. [Figure 12] FIG. 10 is a functional block diagram showing a second example of the configuration of the inspection support device. [Figure 13] 10 is a flowchart schematically illustrating another example of the main processing related to support of the comparison inspection by the control device. [Figure 14] FIG. 10 is a functional block diagram showing a third example of the configuration of the inspection support device. [Figure 15] 10 is a flowchart illustrating another example of pre-processing related to support of a comparison inspection by the control device. [Figure 16] FIG. 10 is a functional block diagram showing a fourth example of the configuration of the examination support device. [Figure 17] 10A and 10B are diagrams for explaining in schematic form the contents of pre-processing related to support of comparative inspection by the control device; [Figure 18] FIG. 10 is a functional block diagram showing a fifth example of the configuration of the examination support device. [Figure 19] 10 is a flowchart schematically illustrating yet another example of pre-processing related to support of a comparison inspection by a control device. [Figure 20] 10A and 10B are diagrams for explaining in schematic form the contents of pre-processing related to support of comparative inspection by the control device; [Figure 21] 10A and 10B are diagrams for explaining in schematic form the contents of pre-processing related to support of comparative inspection by the control device; [Figure 22] 10 is a flowchart schematically illustrating yet another example of the main processing related to support of the comparison inspection by the control device. [Figure 23] FIG. 10 is a diagram schematically illustrating another example of an inspection support system. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment will be described with reference to the drawings.

[0012] [Example of an inspection support system] First, an example of an examination support system 1 according to this embodiment will be described with reference to FIGS.

[0013] FIG. 1 is a schematic diagram showing an example of an inspection support system 1 according to this embodiment. FIGS. 2 to 4 are diagrams showing specific examples of large structures to be inspected by the inspection support system 1. Specifically, FIG. 2 is a side view of a shovel as an example of a large structure to be inspected by the inspection support system 1. FIG. 3 is a side view of a mobile crane (crawler crane) as another example of a large structure to be inspected by the inspection support system 1. FIG. 4 is a side view of a continuous unloader as yet another example of a large structure to be inspected by the inspection support system 1.

[0014] The inspection support system 1 supports a user in inspecting differences between large structures to be compared (hereinafter referred to as "comparison inspection" for convenience).

[0015] Large structures include, for example, large machinery. Large machinery includes, for example, work machines such as excavators, mobile cranes (crawler cranes), and continuous unloaders, as shown in Figs. 2 to 4. Large structures also include, for example, large plants (factory buildings and factory equipment). Large plants include, for example, power plant buildings and equipment, steelworks buildings and equipment, and the like.

[0016] Comparative inspections include, for example, inspecting changes between the same large structure at different times as the object of comparison, that is, inspecting changes in the large structure over time. Changes in the large structure over time include, for example, rust, deformation, cracks, loose screws, missing parts, modifications, discoloration, and relative displacement of the building relative to its surroundings due to earthquakes, etc. Comparative inspections also include, for example, inspecting differences between different large structures of the same type (same design) as the object of comparison. Differences between different large structures of the same type (same design) include, for example, differences in shape or color that exceed expected manufacturing errors, the presence or absence of missing parts, etc.

[0017] The scope of the comparative inspection may be the entire large-scale structure or a part of it. The following description will be focused on the case where the scope of the comparative inspection is the entire large-scale structure.

[0018] The inspection support system 1 includes an inspection support device 100 and a sensor device 200.

[0019] The test support device 100 (an example of an information processing device) supports a comparison test by a user.

[0020] The test support device 100 may be, for example, a terminal device (user terminal) used by a user. The user terminal may be, for example, a stationary terminal device such as a desktop computer terminal. The user terminal may also be, for example, a portable (portable) terminal device (mobile terminal) such as a smartphone, a tablet terminal, or a laptop computer terminal.

[0021] The test support device 100 includes a control device 110 , a communication device 120 , an input device 130 , and a display device 140 .

[0022] The control device 110 controls support for the comparative inspection by the user. Based on a group of sensing data regarding the shape of one large structure to be compared and a group of sensing data regarding the shape of the other large structure to be compared, which are acquired from the sensor device 200, the control device 110 provides information to support the comparative inspection through the display device 140.

[0023] The functions of the control device 110 may be realized by any hardware or any combination of hardware and software. The control device 110 is mainly configured with a computer including, for example, a central processing unit (CPU), a memory device, a non-volatile auxiliary storage device, and various input / output interface devices. The control device 110 realizes various functions by, for example, loading programs installed in the auxiliary storage device into the memory device and executing the programs on the CPU. The memory device is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The auxiliary storage device is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The interface device includes, for example, an external interface for connecting to a recording medium. This allows the inspection support device 100 (control device 110) to, for example, retrieve various programs from a recording medium via the external interface and install them in the auxiliary storage device.

[0024] The communication device 120 communicates with an external device (e.g., a sensor device 200) through a predetermined communication line. This allows the test assistance device 100 to exchange data with the external device through the communication device 120. The test assistance device 100 can also retrieve various programs from the external device and install them in the control device 110 (auxiliary storage device). The predetermined communication line may be, for example, a one-to-one communication line. The predetermined communication line may also include, for example, a local area network (LAN) within a facility where a user performs work related to the comparison test. The local network may be wired, wireless, or may include both. The predetermined communication line may also include a wide area network (WAN) outside the facility where a user performs work related to the comparison test. The wide area network may include, for example, a mobile communication network terminated at a base station, a satellite communication network using a communication satellite, the Internet, or the like. The predetermined communication line may also include, for example, a short-range communication line based on a predetermined wireless communication standard such as Bluetooth (registered trademark) or WiFi.

[0025] The function of the communication device 120 may be incorporated into the control device 110 as an interface device for the control device 110.

[0026] The input device 130 accepts various inputs from a user of the test assistance system 1. The input device includes, for example, an operation input device such as a keyboard, a mouse, a joystick, a touch panel, a touch pad, a button, a toggle, or a lever. The input device 130 may also include, for example, a voice input device or a gesture input device that accepts voice input or gesture input from the user. The input device 130 may also include, for example, a biometric input device that allows biometric input from the user through fingerprint authentication, iris authentication, or the like. A signal corresponding to the input content from the user accepted by the input device 130 is taken into the control device 110.

[0027] The display device 140 displays an information image relating to the comparison inspection under the control of the control device 110. The display device 140 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0028] The sensor device 200 (an example of a sensor) acquires sensing data (hereinafter simply referred to as "sensing data") related to the shape of a large structure that is the target of a comparison inspection. The sensor device 200 is, for example, an imaging device capable of outputting captured images (image data) of the large structure. The imaging device may include, for example, a monocular camera, a stereo camera, a depth camera, etc. The sensor device 200 may also be a distance sensor capable of acquiring point cloud data corresponding to the shape of the large structure based on the sensor device 200, such as a LIDAR (Light Detecting and Ranging), a millimeter-wave radar, or an ultrasonic sensor. The following description will focus on the case where the sensing data is image data. The sensor device 200 may also be an information device (smart device) equipped with an imaging device, a distance sensor, etc. Information devices include, for example, smartphones and tablet terminals. The sensor device 200 may also be a drone equipped with an imaging device, a distance sensor, etc.

[0029] The sensor device 200 is communicatively connected to the inspection support device 100 via a predetermined communication line, and a collection of sensing data of a large structure acquired by the sensor device 200 (hereinafter referred to as "sensing data group") is captured into the inspection support device 100. For example, the sensor device 200 may be connected to the inspection support device 100 by a user via a cable (one-to-one communication line), thereby capturing the sensing data group into the inspection support device 100. Alternatively, for example, the sensor device 200 may transmit the sensing data to the inspection support device 100 via a mobile communication network or WiFi, and the inspection support device 100 may receive the sensing data via the communication device 120, thereby capturing the sensing data into the inspection support device 100.

[0030] Furthermore, the sensing data group acquired by the sensor device 200 may be imported into the test support apparatus 100 via a portable storage medium. Portable storage media include, for example, magneto-optical disks, optical disks, removable hard disks, flash memories, etc. Optical disks include, for example, writable media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), and BDs (Blu-ray (registered trademark) Discs). Flash memories include, for example, USB (Universal Serial Bus) flash drives (USB memory), SD memory cards, etc.

[0031] [Example 1 of a method for supporting comparative testing] Next, a first example of a method for supporting a comparison test will be described with reference to FIGS.

[0032] <Functional configuration of the examination support device> FIG. 5 is a functional block diagram showing a first example of the configuration of the test support device 100. As shown in FIG.

[0033] 5, the control device 110 of the inspection support device 100 includes an image data storage unit 1101, a shape estimation unit 1102, an overall shape data storage unit 1103, a link processing unit 1104, a link DB (Data Base) 1105, and a comparison display unit 1106. These functions are realized, for example, by loading a program installed in an auxiliary storage device into a memory device and having the CPU execute the program, or by defining a predetermined storage area in the auxiliary storage device.

[0034] The image data storage unit 1101 stores a group of image data of large structures that are acquired from the sensor devices 200 via the communication device 120 .

[0035] For example, image data groups of large structures captured at different times are stored in the image data storage unit 1101 in a form distinguished by addresses, etc. This allows the control device 110 (shape estimation unit 1102) to appropriately distinguish and read out the image data groups corresponding to the two large structures to be compared when a comparative inspection is performed.

[0036] Furthermore, for example, the image data storage unit 1101 may store a plurality of image data sets acquired at different times by the sensor device 200 without distinguishing them into image data groups. In this case, the control device 110 may distinguish the image data groups based on date information or the like included in the metadata for each image data set.

[0037] The shape estimation unit 1102 (an example of an estimation unit) estimates (reproduces) the overall shape (three-dimensional shape) of each large structure to be compared based on the image data sets of the respective large structures stored in the image data storage unit 1101 in response to a predetermined input (request) from the user via the input device 130. Specifically, the shape estimation unit 1102 generates and outputs three-dimensional model data (hereinafter referred to as "overall shape data") corresponding to the overall shape of the target large structure based on the image data sets of the target large structure. For example, the shape estimation unit 1102 may use a known photogrammetry technique to create a three-dimensional model representing the overall shape of the large structure based on the image data sets of the target large structure.

[0038] The image data group of the target large structure may include multiple image data of the same part acquired when at least one of the position and orientation of the sensor device 200 is different. For example, the image data group of the target large structure may include data of multiple still images included in a video acquired while the sensor device 200 is moving. This makes it easier for the shape estimation unit 1102 to estimate the overall shape based on the image data of the same part. Therefore, the control device 110 can improve the accuracy of estimating the overall shape of the target large structure.

[0039] When the range of the comparison inspection is a part of a large structure, the shape estimation unit 1102 may estimate (reproduce) the three-dimensional shape of an area corresponding to that part, rather than the entire shape of the large structure. The same applies to the shape estimation unit 1102A described below.

[0040] The overall shape data storage unit 1103 stores the overall shape data of the large structure output by the shape estimation unit 1102.

[0041] The link processor 1104 (an example of a sorting unit) associates each image data included in the original image data group of the overall shape data with a portion of the overall shape data. Specifically, for each of the multiple portions defined in the overall shape data, the link processor 1104 may associate the portion of the overall shape data with the image data in the image data group. For example, the link processor 1104 may divide the overall shape data into any number of portions using a predetermined method, and associate each portion of the overall shape data with the image data in the image data group. Alternatively, for example, the link processor 1104 may divide the overall shape data into multiple portions corresponding to multiple inspection target portions of a large structure defined in advance, and associate each portion of the overall shape data with the image data in the image data group. In this case, the link processor 1104 may divide the overall shape data into multiple portions using, for example, design data of the large structure and multiple inspection target portions defined in the design data.

[0042] The link DB 1105 stores record data indicating the association between the parts of the overall shape data created by the link processing unit 1104 and the image data of the original image data group.

[0043] In response to a predetermined input (request) from the user via the input device 130, the comparison display unit 1106 (an example of a display unit) displays image data of common parts of both large structures being compared on the display device 140 in a manner that allows comparison. Specifically, the comparison display unit 1106 may use the link DB 1105 to display, side by side, the image data associated with corresponding parts (common parts) in the overall shape data of one and the other large structures being compared on the display device 140. A common part means the same part, or, if there are no exactly identical parts, a part that exists within a predetermined proximity.

[0044] For example, the comparison display unit 1106 may be configured such that when image data is selected from the image data group of one large structure by user input to the input device 130, image data of common parts from the image data group of the other large structure are displayed side by side.

[0045] Furthermore, for example, the comparison display unit 1106 may display on the display device 140 some or all of the combinations of image data associated with corresponding portions of the overall shape data of both large structures being compared in a manner that allows for scrolling and viewing.

[0046] Furthermore, for example, the comparison display unit 1106 may display image information (hereinafter, "large structure images") representing the overall shapes of both large structures to be compared on the display device 140. The large structure images may be, for example, image information that schematically shows the overall shapes of the large structures. The large structure images may also be, for example, images of three-dimensional models corresponding to the overall shape data of both large structures to be compared. In this case, the images of the three-dimensional models displayed on the display device 140 may be still images of the three-dimensional models viewed from a predetermined direction. The images of the three-dimensional models displayed on the display device 140 may also be images whose viewpoint and size can be changed in response to user input via the input device 130, such as the visual interface of a three-dimensional CAD (Computer Aided Design) system. The comparison display unit 1106 may then display on the display device 140 a combination of image data associated with a portion of the overall shape data of the large structures to be compared, which corresponds to an arbitrary portion of the large structure images selected by the user's input via the input device 130.

[0047] <Preparation for comparative testing support> Fig. 6 is a flowchart that schematically shows an example of pre-processing related to support for comparative inspection by the control device 110. Figs. 7 and 8 are diagrams that schematically explain the contents of pre-processing related to support for comparative inspection by the control device 110. Specifically, Figs. 7 and 8 are diagrams that schematically explain the contents of pre-processing related to support for comparative inspection based on image data groups DG1 and DG2 of a first shovel and a second shovel, respectively, as large structures.

[0048] The flowchart of Fig. 6 may be executed, for example, when an image data group (e.g., image data groups DG1 and DG2) is acquired from the sensor device 200 and a selection input to execute pre-processing is made through the input device 130. The flowchart of Fig. 6 may also be executed, for example, when a request to execute pre-processing is made from the user through the input device 130 and the user selects an image data group to be subjected to pre-processing through the input device 130. The same may be true for the flowchart of Fig. 15 described below.

[0049] As shown in FIG. 6, in step S102, the shape estimation unit 1102 estimates the overall shape of the large structure corresponding to the image data group based on the image data group (multiple image data included in the image data group) of the large structure, and outputs the overall shape data.

[0050] 7, the shape estimation unit 1102 creates a three-dimensional model MD1 as overall shape data of the first shovel based on image data IMG11, IMG12, IMG13, IMG14, IMG15, and so on included in the image data group DG1. The image data group DG1 of the first shovel was acquired when the tip of the attachment (bucket), which includes the boom, arm, and bucket, was relatively separated from the machine body (upper rotating body). Therefore, the three-dimensional model MD1 reproduces the external shape (shape of the attachment portion) of the first shovel that corresponds to that state.

[0051] 8, the shape estimation unit 1102 creates a three-dimensional model MD2 as overall shape data of the second shovel based on image data IMG21, IMG22, IMG23, IMG24, IMG25, etc. included in the image data group DG2. The image data group DG2 of the second shovel was acquired when the tip of the attachment (bucket) including the boom, arm, and bucket was in a state relatively close to the machine body (upper rotating body). Therefore, the three-dimensional model MD2 reproduces the external shape (shape of the attachment portion) of the second shovel corresponding to that state.

[0052] Returning to FIG. 6, when the control device 110 completes the process of step S102, the process proceeds to step S104.

[0053] In step S104, the link processing unit 1104 associates each image data in the image data group with a part in the overall shape data.

[0054] For example, as shown in FIG. 7, the link processing unit 1104 associates image data IMG11, IMG12, IMG13, IMG14, and IMG15 of the image data group DG1 with parts P13, P11, P12, P15, and P14 of the three-dimensional model MD1, respectively.

[0055] Also, for example, as shown in FIG. 8, the link processing unit 1104 associates image data IMG21, IMG22, IMG23, IMG24, and IMG25 of the image data group DG2 with parts P22, P24, P23, P25, and P21 of the three-dimensional model MD2, respectively.

[0056] In addition to the process of step S104, the control device 110 may associate each image data in the image data group with other information. For example, the control device 110 may associate, for each image data in the image data group, image data of a target with information on the specifications of the sensor device 200 (image capture device), such as the angle of view. Furthermore, for each image data in the image data group, the control device 110 may associate, for example, image data of a target with information on the distance from the sensor device 200 (image capture device) to the subject (large structure).

[0057] Returning to FIG. 6, when the control device 110 completes the process of step S104, it ends the process of this flowchart.

[0058] Thus, in this example, the control device 110 can reproduce the entire three-dimensional shape of the large-scale structure being compared based on two groups of image data of the large-scale structure being compared, and organize the two groups of image data so that common parts of the large-scale structure being compared can be compared.

[0059] <Main process for supporting comparative testing> FIG. 9 is a flowchart outlining an example of main processing related to support of a comparative inspection by the control device 110. FIG. 10 is a diagram showing an example of a combination of image data of common parts of large structures to be compared, extracted from image data groups. Specifically, FIG. 10 is a diagram showing an example of a combination of image data of common parts of a first shovel and a second shovel to be compared, extracted from image data groups DG1 and DG2 (see FIGS. 7 and 8). FIG. 11 is a diagram showing an example of a combination of image data of common parts of large structures to be compared, displayed on the display device 140. Specifically, FIG. 11 is a diagram showing an example of a combination of image data of common parts of a first shovel and a second shovel to be compared, displayed on the display device 140.

[0060] 9 may be executed when two image data groups corresponding to large structures to be compared are selected, for example, when a user inputs a request to execute main processing related to supporting a comparative inspection via the input device 130. The same may be true for the flowchart of FIG. 13 described below.

[0061] As shown in FIG. 9, in step S202, the comparison display unit 1106 associates common parts of two sets of overall shape data corresponding to two sets of image data to be compared.

[0062] For example, if a large structure does not have any moving parts, the comparison display unit 1106 compares the shapes and positions (e.g., coordinates on the overall shape data) of the two overall shape data to match common parts of the two overall shape data.

[0063] Furthermore, for example, if a large structure has movable parts, the comparison display unit 1106 refers to the design data of the large structure and matches the common parts of the movable parts of the two overall shapes based on the shape, position, range of motion, etc. of the movable parts.

[0064] As shown in FIGS. 7 and 8, the attachments (boom, arm, and bucket) of the first and second shovels are movable parts, as described above. Therefore, the three-dimensional model MD1 corresponding to the first shovel and the three-dimensional model MD2 corresponding to the second shovel have completely different shapes of attachments. Therefore, the comparison display unit 1106 can associate common parts of the first and second shovels by referring to the design data. Specifically, the comparison display unit 1106 can associate a part P14 of the attachment (arm) in the three-dimensional model MD1 with a part P23 of the attachment (arm) in the three-dimensional model MD2 as a common part.

[0065] Returning to FIG. 9, when the control device 110 completes the process of step S202, the process proceeds to step S204.

[0066] In step S204, the comparison display unit 1106 sets a common direction (hereinafter referred to as "reference direction") and scale (hereinafter referred to as "reference scale") for each common part of the two overall shape data associated in step S202.

[0067] For example, the reference direction is defined as the direction as seen from each of the common parts of the two overall shape data. That is, the reference direction is defined on a coordinate system fixed to each of the common parts of the two overall shape data. As a result, for example, if the common part corresponds to a movable part of a large structure to be compared, the reference direction can also be moved in accordance with the movement of the movable part of the large structure. Therefore, even if the shapes (postures) of the attachments of the movable parts are completely different, as in the first and second shovels described above, for example, the reference direction as seen from the common part can be aligned between the two overall shape data.

[0068] When the process of step S204 is completed, the control device 110 proceeds to step S206.

[0069] In step S206, the comparison display unit 1106 extracts, for each common part of the two overall shape data, image data associated with the common part from each of the two image data groups.

[0070] For example, as shown in FIG. 10, the comparison display unit 1106 extracts image data IMG15 and IMG23 from the image data groups DG1 and DG2, which correspond to a part P14 of the three-dimensional model MD1 and a part P23 of the three-dimensional model MD2, respectively, which correspond to a common part.

[0071] Returning to FIG. 9, when the process of step S206 is completed, the control device 110 proceeds to step S208.

[0072] In step S208, the comparison display unit 1106 displays the extracted combination of image data for each common part of the two overall shape data on the display device 140 in accordance with the reference direction and the reference scale.

[0073] 11, the comparison display unit 1106 displays the image data IMG15 and IMG23 with the same reference direction and scale. Specifically, in this example, the image direction and size of the image data IMG15 are corrected to align the reference direction and scale of the image data IMG15 and IMG23.

[0074] Returning to FIG. 9, when the process of step S208 is completed, the control device 110 ends the process of this flowchart.

[0075] In this way, in this example, based on the two image data groups organized as described above, the control device 110 can display on the display device 140 a comparison of combinations of image data from the two image groups corresponding to common parts of the large structures being compared. This allows the user to compare the two image data corresponding to common parts of the large structures being compared, and to inspect changes over time in the same large structure or differences between large structures of the same design (same type).

[0076] [Second example of a method for supporting comparative testing] Next, a second example of the method for supporting a comparison test will be described with reference to Figures 12 and 13. The following description will focus on the differences from the first example described above, and descriptions of the same or corresponding content may be simplified or omitted.

[0077] <Functional configuration of the examination support device> FIG. 12 is a functional block diagram showing another example of the configuration of the test support device 100.

[0078] 12, the control device 110 of the inspection support device 100 includes an image data storage unit 1101, a shape estimation unit 1102, an overall shape data storage unit 1103, a link processing unit 1104, a link DB 1105, a comparison display unit 1106, and a difference detection unit 1107. These functions are realized, for example, by loading a program installed in an auxiliary storage device into a memory device and having the CPU execute the program, or by defining a predetermined storage area in the auxiliary storage device.

[0079] The difference detection unit 1107 (an example of a detection unit) detects differences between combinations of image data corresponding to common regions extracted from two sets of image data corresponding to large structures to be compared.

[0080] For example, the difference detection unit 1107 may apply, for example, a classifier based on known image processing technology or machine learning to recognize (detect) differences of a type defined for each part (for example, scratches, dents, the mounting posture of a specified part, the presence or absence of a specified part, etc.).

[0081] Furthermore, for example, the difference detection unit 1107 may preliminarily label all of the image data in each of the two image data groups corresponding to the large structures to be compared, indicating features related to the differences (i.e., assigning attribute data related to the differences). Labels indicating features related to the differences may include, for example, a label indicating the presence of a scratch, a label indicating the presence of a dent, or a label indicating a missing part. Specifically, the difference detection unit 1107 may apply, for example, a known image processing technique or a classifier based on machine learning to recognize the presence or absence of features corresponding to the target labels for all of the image data in each of the two image data groups. If a feature corresponding to the label is found, the difference detection unit 1107 assigns the target label as metadata to the image data. This allows the difference detection unit 1107 to detect differences between combinations of image data corresponding to common parts based on the differences in the assigned labels.

[0082] Furthermore, the difference detection unit 1107 may select (group) all common parts of the two overall shape data corresponding to the large structures to be compared based on at least one of the content of the difference and the degree of the difference. This allows the control device 110 (comparison display unit 1106) to limit a specific group of all common parts and display combinations of image data corresponding to the common parts on the display device 140. Therefore, the user can select a specific group using the input device 130 and limit the parts for which combinations of image data are to be displayed, such as parts where a specific difference occurs or parts where the degree of difference exceeds a predetermined standard. This allows the control device 110 to improve user convenience and the efficiency of the comparison inspection.

[0083] <Main process for supporting comparative testing> FIG. 13 is a flowchart schematically showing another example of the main processing related to the support of the comparison inspection by the control device 110.

[0084] As shown in FIG. 13, steps S302 to S306 are the same as steps S202 to S206 in FIG. 9, and therefore a description thereof will be omitted.

[0085] When the process of step S306 is completed, the control device 110 proceeds to step S308.

[0086] In step S308, the difference detection unit 1107 detects the difference between the combinations of image data extracted in step S306 for each common part of the two overall shape data.

[0087] When the process of step S308 is completed, the control device 110 proceeds to step S310.

[0088] In step S310, the comparison display unit 1106 displays on the display device 140 a combination of image data for each part where a difference has been detected, in accordance with the reference direction and the reference scale.

[0089] For example, the comparison display unit 1106 may display a combination of image data in a manner that emphasizes the differences between them. Specifically, the comparison display unit 1106 may display a marker on a portion of image data where a difference occurs between one image data and the other image data. Furthermore, if one image data contains a feature (e.g., a scratch or a dent) that does not exist in the other image data, the comparison display unit 1106 may perform image processing to make the feature of the other image data more noticeable (e.g., increase the amount of the scratch or dent) and then display the image.

[0090] Furthermore, for example, similar to the example described above, the comparison display unit 1106 may also display an image of the large structure on the display device 140. Then, the comparison display unit 1106 may display information about the difference (for example, summary information indicating the content or degree of the difference) in association with the portion of the large structure image corresponding to the portion where the difference was detected.

[0091] When the process of step S310 is completed, the control device 110 ends the process of this flowchart.

[0092] In this way, in this example, the control device 110 detects differences between common portions of the large structures being compared based on the two image data groups organized as described above. The control device 110 then displays image data from the two image data groups for the common portions of the large structures being compared where differences exist on the display device 140 so that the image data can be compared. This allows the user to check only the combinations of image data where differences exist. Therefore, the control device 110 can improve user convenience and the efficiency of the comparison inspection.

[0093] [Third example of a method for supporting comparative testing] Next, a third example of a method for supporting a comparison test will be described with reference to Figures 14 and 15. The following description will focus on differences from the first example and other examples, and explanations of the same or corresponding content may be simplified or omitted.

[0094] <Functional configuration of the examination support device> FIG. 14 is a functional block diagram showing a third example of the configuration of the test support device 100. As shown in FIG.

[0095] 14, the control device 110 of the inspection support device 100 includes an input condition determination unit 1100, an image data storage unit 1101, a shape estimation unit 1102, an overall shape data storage unit 1103, a link processing unit 1104, a link DB 1105, and a comparison display unit 1106. These functions are realized, for example, by loading a program installed in an auxiliary storage device into a memory device and having the CPU execute the program, or by defining a predetermined storage area in the auxiliary storage device.

[0096] In this example (FIG. 14), the input condition determining unit 1100 is added to the example described above (FIG. 6), but the input condition determining unit 1100 may be added to the other example described above (FIG. 12).

[0097] The input condition determination unit 1100 (an example of a determination unit) determines whether or not a group of image data of a large structure, which is acquired from the sensor device 200 via the communication device 120, satisfies a predetermined input condition. The predetermined input condition is, for example, that the acquired image data group includes image data of all specified parts of the large structure (for example, all parts to be inspected) (an example of predetermined parts).

[0098] If the input condition determination unit 1100 determines that the input conditions are satisfied, the image data group is stored in the image data storage unit 1101. On the other hand, if the input condition determination unit 1100 determines that the input conditions are not satisfied, the image data group is discarded. Also, if the input condition determination unit 1100 determines that the input conditions are not satisfied, the image data group may be stored in the image data storage unit 1101 as an image data group that cannot be used for comparison testing. Also, the user may be requested to provide additional data to satisfy the input conditions via the display device 140. In this case, when additional data is acquired from the sensor device 200 and the image data group that combines the already acquired data and the additional data satisfies the input conditions, the image data group may be stored in the image data storage unit 1101 as image data that can be used for comparison testing.

[0099] <Preparation for comparative testing support> FIG. 15 is a flowchart schematically illustrating another example of pre-processing related to support for comparison inspection.

[0100] 15, in step S402, the input condition determination unit 1100 determines whether the imported image data set includes image data of all specified portions of the large structure. If the imported image data set includes image data of all specified portions of the large structure, the input condition determination unit 1100 proceeds to step S404; otherwise, the processing of this flowchart ends.

[0101] Steps S404 and S406 are the same as steps S102 and S204 in FIG. 6, and therefore a description thereof will be omitted.

[0102] When the process of step S406 is completed, the control device 110 ends the process of this flowchart.

[0103] In this way, in this example, the control device 110 can determine whether image data of a specified portion of a large structure to be compared is included in the image data group acquired from the sensor device 200. Therefore, the control device 110 can prompt the user to prepare an image data group that will allow for appropriate comparison inspection.

[0104] [Fourth example of a method for supporting comparative testing] Next, a fourth example of the method for supporting a comparison test will be described with reference to Figures 16 and 17. The following description will focus on the differences from the first example and other examples described above, and descriptions of the same or corresponding content may be simplified or omitted.

[0105] <Functional configuration of the examination support device> FIG. 16 is a functional block diagram showing a fourth example of the functional configuration of the test support device 100. As shown in FIG.

[0106] As shown in FIG. 16, the control device 110 of the inspection support device 100 includes an image data storage unit 1101, a shape estimation unit 1102A, an overall shape data storage unit 1103A, a link processing unit 1104A, a link DB 1105A, and a comparison display unit 1106A.

[0107] The shape estimation unit 1102A (an example of an estimation unit) estimates (reproduces) the approximate overall shapes (three-dimensional shapes) of both large structures to be compared by combining the image data groups of both large structures to be compared, which are stored in the image data storage unit 1101, in response to a predetermined input (request) from the user via the input device 130. That is, based on the image data groups of both large structures to be compared, the shape estimation unit 1102A estimates the overall shape (three-dimensional shape) of the same large structure (hereinafter referred to as "the same large structure" for convenience) when both large structures are assumed to be the same large structure.

[0108] For example, if a large structure has movable parts, the shape estimation unit 1102A refers to the design data of the large structure, allowing the shape estimation unit 1102A to estimate the overall shape of the same large structure while correcting for differences between the three-dimensional shapes of one large structure based on the image data group and the other large structure based on the image data group, which differences arise due to the influence of the movable parts.

[0109] In this example, it is sufficient that the overall shape of the same large structure can be estimated by combining the image data group of one large structure to be compared with the image data group of the other large structure. In other words, the image data group of one large structure to be compared and the image data group of the other large structure do not need to be a dense collection of image data covering the entire range of the large structure, so that the overall shape of the same large structure can be estimated using each image data group alone. For example, the image data group of one large structure may be a dense collection of image data covering the entire range of the one large structure, while the image data group of the other large structure may be a sparse collection of image data covering a limited range of the other large structure. Furthermore, for example, the overall shape data of the same large structure may be estimated using the image data group of one large structure, which is a dense collection of image data covering the entire range of the one large structure, and only one image data of the other large structure to be compared. This allows, for example, a user to perform a comparison inspection using only image data of a specific portion to be compared with one large structure.

[0110] The overall shape data storage unit 1103A stores overall shape data of the same large structure estimated (generated) by the shape estimation unit 1102A.

[0111] The link processing unit 1104A (an example of a sorting unit) associates the original image data groups of the overall shape data of the same large structure (respective image data groups of both large structures being compared) with parts of the overall shape data of the same large structure.

[0112] In the link DB 1105A, record data indicating the association between the parts of the overall shape data by the link processing unit 1104 and the image data of the original image data group is registered.

[0113] Similar to the first example described above, the comparison display unit 1106A (an example of a display unit) displays image data of the same portion (common portion) of both large structures being compared in a manner that allows comparison in response to a predetermined input (request) from the user on the display device 140. Specifically, the comparison display unit 1106A may use the link DB 1105A to display side by side on the display device 140 the image data of both large structures being compared that are associated with the common portion of the overall shape data of the same large structure.

[0114] <Preparation for comparative testing support> Fig. 17 is a diagram that schematically illustrates the contents of pre-processing related to support for comparative inspection by the control device 110. Specifically, Fig. 17 is a diagram that schematically illustrates the contents of pre-processing related to support for comparative inspection based on image data groups DG3 and DG4 of the first and second shovels as large structures.

[0115] In this example, the test support device 100 (control device 110) can perform pre-processing related to support of the comparison test by the same processing as in the first example (FIG. 6) described above.

[0116] As shown in FIG. 6, in step S102, the shape estimation unit 1102A estimates the overall shape of the same large structure based on the image data groups of both large structures to be compared, and outputs overall shape data.

[0117] For example, as shown in FIG. 17, the shape estimation unit 1102A creates a three-dimensional model MD3 as overall shape data of the same large structure (the same shovel) corresponding to the first shovel and the second shovel, based on image data IMG31, IMG32, IMG33, IMG34, IMG35, ... and image data IMG41, IMG42, IMG43, IMG44, IMG45, ... contained in the image data groups DG3 and DG4, respectively.

[0118] Returning to FIG. 6, when the control device 110 completes the process of step S102, the process proceeds to step S104.

[0119] In step S104, the link processor 1104A associates each image data in the image data groups of both large structures to be compared with parts of the same large structure.

[0120] 16, the link processing unit 1104A associates image data IMG31, IMG32, IMG33, IMG34, and IMG35 of the image data group DG3 with regions P33, P31, P32, P35, and P34 of the three-dimensional model MD3, respectively. The link processing unit 1104A also associates image data IMG41, IMG42, IMG43, IMG44, and IMG45 of the image data group DG4 with regions P33, P31, P32, P35, and P34 of the three-dimensional model MD3, respectively. This allows the user to compare common regions (same regions) of the first and second shovels being compared, for example, by comparing the image data of the image data groups DG3 and DG4 that are associated with the same regions P31 to P35.

[0121] Returning to FIG. 6, when the control device 110 completes the process of step S104, it ends the process of this flowchart.

[0122] Thus, in this example, the control device 110 can reproduce the entire three-dimensional shape of the same large structure based on two groups of image data of the large structure being compared, and organize the two groups of image data so that common parts of the large structure being compared can be compared.

[0123] <Main process for supporting comparative testing> In this example, the inspection support device 100 (control device 110) can perform pre-processing related to supporting the comparison inspection by omitting step S202 from the processing of the first example (FIG. 9) described above. This is because, in this example, image data from the two image data groups of the large structures to be compared have already been associated with each other for the common parts of the same overall shape data, as described above.

[0124] In this way, in this example, similar to the first example and the like, the control device 110 can display, on the display device 140, a comparison of the combinations of image data of the two image groups corresponding to common parts of the large structures to be compared, based on the two image data groups organized as described above. This allows the user to compare the two image data corresponding to common parts of the large structures to be compared, and to inspect changes over time in the same large structure or differences between large structures of the same design (same type).

[0125] [5th ​​example of a method for supporting comparative testing] Next, a fifth example of the method for supporting a comparison test will be described with reference to Figures 18 to 22. The following description will focus on the differences from the first example and other examples described above, and descriptions of the same or corresponding content may be simplified or omitted.

[0126] In this example, one of the large structures to be compared is fixed, and only the other large structure to be compared can be changed. When the comparative inspection is an inspection of time-series changes in the large structure, the one large structure to be compared is, for example, the large structure at the time of completion (when new).

[0127] <Functional configuration of the examination support device> FIG. 18 is a functional block diagram showing a fifth example of the configuration of the test support device 100. As shown in FIG.

[0128] As shown in FIG. 18, the control device 110 of the inspection support device 100 includes a shape estimation unit 1102B, a comparison data storage unit 1108, a reference data storage unit 1109, a link processing unit 1104B, a link DB 1105B, and a comparison display unit 1106B.

[0129] The shape estimation unit 1102B estimates (reproduces) the three-dimensional shape of the range corresponding to the image data or image data group of the other large structure to be compared, based on the image data or image data group of the other large structure.

[0130] The shape estimation unit 1102B may be omitted.

[0131] Image data and three-dimensional shape data of the other large structure to be compared are stored in the comparison data storage unit 1108. Hereinafter, the image data, three-dimensional shape data, etc. of the other large structure to be compared may be referred to collectively or individually as "comparison data."

[0132] The reference data storage unit 1109 stores in advance a group of image data and three-dimensional shape data (overall shape data) of the overall shape of one of the large structures to be compared. For each piece of image data included in the image data group serving as reference data, the corresponding portion of the overall shape data serving as reference data is identified in advance, and the reference data storage unit 1109 also stores record data indicating this correspondence. Hereinafter, the image data of one of the large structures to be compared, the image data group as a collection of such data, the overall shape data, etc. may be referred to collectively or individually as "reference data."

[0133] The overall shape data of one large structure to be compared as reference data may be estimated (generated) based on image data of the one large structure, or may be three-dimensional computer-aided design (CAD) data of the one large structure. In the former case, the overall shape data of the one large structure may be estimated (generated) by the shape estimation unit 1102B, or may be acquired from outside the inspection support device 100 via the communication device 120. In the latter case, the three-dimensional CAD data of the one large structure may be data generated, for example, during the design or development of the large structure, and may be acquired from outside the inspection support device 100 via the communication device 120.

[0134] The link processing unit 1104B (an example of a sorting unit) associates the image data group and overall shape data (reference data) of one large structure with the image data (group) and three-dimensional shape data (comparison data) of the other large structure being compared, so that common parts of both large structures being compared can be compared.

[0135] Specifically, the link processing unit 1104B may employ a known matching technique to match the image data group and overall shape data (reference data) of one large structure with the image data of the other large structure. The link processing unit 1104B may also match the image data group and overall shape data (reference data) of one large structure with the three-dimensional shape data of the other large structure. The link processing unit 1104B may also optimize the matching results. This allows the link processing unit 1104B to associate the image data of the other large structure with a common portion in the overall shape data of the one large structure and the corresponding other large structure. The link processing unit 1104B may also associate the image data of the other large structure with the image data of the one large structure that is associated with a common portion in the overall shape data of the one large structure. The link processing unit 1104B can also associate the three-dimensional shape data of the other large structure with a portion in the overall shape data of one large structure that is common to the other large structure corresponding to the three-dimensional shape data of that large structure. The link processing unit 1104B can also associate the three-dimensional shape data of the other large structure with image data of one large structure that is associated with a portion in the overall shape of the one large structure that is common to the other large structure.

[0136] In the link DB 1105B, record data indicating the association by the link processing unit 1104B is registered.

[0137] The comparison display unit 1106B (an example of a display unit) displays image data or three-dimensional shape data corresponding to a common portion of both large structures to be compared on the display device 140 in a comparable manner in response to a predetermined input (request) from the user via the input device 130. Furthermore, the comparison display unit 1106B may display one image data and the other three-dimensional shape data, or one three-dimensional shape data and the other image data, corresponding to a common portion of both large structures to be compared, on the display device 140 in a comparable manner.

[0138] <Preparation for comparative testing support> Fig. 19 is a flowchart that schematically illustrates yet another example of pre-processing related to support for a comparative inspection by the control device 110. Figs. 20 and 21 are diagrams that schematically explain the contents of pre-processing related to support for a comparative inspection by the control device 110. Specifically, Fig. 20 is a diagram that schematically explains the contents of pre-processing related to support for a comparative inspection based on reference data REF of a first shovel as one large structure and image data IMG_C1 of a second shovel as the other large structure. Fig. 21 is a diagram that schematically explains the contents of pre-processing related to support for a comparative inspection based on reference data REF of a first shovel as one large structure and image data IMG_C1 and a three-dimensional model MD_C1 (comparison data) of the other large structure.

[0139] In the examples of FIGS. 20 and 21, the reference data REF includes an image data group DG_R of the first shovel and a three-dimensional model MD_R corresponding to overall shape data. Furthermore, in this example, image data IMG_R1, IMG_R2, IMG_R3, IMG_R4, and IMG_R5 included in the reference data REF are pre-associated with portions P_R3, P_R1, P_R2, P_R5, and P_R4 of the three-dimensional model MD_R, respectively. Furthermore, the comparison data is image data IMG_C1 in the example of FIG. 20, and includes the image data IMG_C1 and a three-dimensional model MD_C1 including a portion (arm portion) of the second shovel corresponding to the image data IMG_C1 in the example of FIG. 21. The three-dimensional model MD_C1 corresponds to the arm and bucket portions of the second shovel.

[0140] As shown in FIG. 19, in step S502, the link processing unit 1104B associates the comparison data (for example, image data of the other large structure) with a portion of the overall shape (overall shape data) of the reference data.

[0141] 20, the link processing unit 1104B may match image data IMG_C1 with a portion P_R4 of the three-dimensional model MD_R by matching the image data IMG_C1 with the three-dimensional model MD_R. The link processing unit 1104B may also match image data IMG_C1 with image data IMG_R5 of the reference data REF (image data group DG_R). This is because image data IMG_R5 of the reference data REF (image data group DG_R) has been matched in advance with the portion P_R4 of the three-dimensional model MD_R with which the image data IMG_C1 has been matched.

[0142] 21, the link processing unit 1104B may associate the three-dimensional model MD_C1 of the second shovel with the three-dimensional model MD_R1 by matching the three-dimensional model MD_C1 of the second shovel with the three-dimensional model MD_R of the first shovel. The three-dimensional model MD_C1 is a part of the three-dimensional model MD_R representing the overall shape of the first shovel, and corresponds to a portion of the first shovel common to the three-dimensional model MD_C1 (the arm and bucket portion). The link processing unit 1104B may also associate image data IMG_C1 corresponding to a portion P_C1 of the three-dimensional model MD_C1 with a portion P_R4 of the three-dimensional model MD_R1 that is common to the portion P_C1. The link processing unit 1104B may also associate the image data IMG_C1 of the second shovel with image data IMG_R5 of the first shovel that corresponds to the portion P_R4 of the three-dimensional model MD_R1.

[0143] Returning to FIG. 19, when the process of step S502 is completed, the control device 110 ends the process of the flowchart.

[0144] In this way, in this example, the control device 110 can use the reference data of one of the large structures being compared to organize the relationship between the reference data of one large structure and the comparison data of the other large structure so that common parts of the large structures being compared can be compared.

[0145] <Main process for supporting comparative testing> FIG. 22 is a flowchart schematically showing yet another example of pre-processing related to support of a comparison inspection by the control device 110.

[0146] As shown in FIG. 22, step S602 is the same as the processing in step S204 in FIG. 9, and therefore a description thereof will be omitted.

[0147] When the process of step S602 is completed, the control device 110 proceeds to step S604.

[0148] In step S604, the comparison display unit 1106B uses the link DB 1105 to extract reference data (image data or three-dimensional shape data) of the same region as the comparison data (image data or three-dimensional shape data).

[0149] When the process of step S604 is completed, the control device 110 proceeds to step S606.

[0150] In step S606, the combination of the extracted reference data and comparison data for each region is displayed on the display device 140 in accordance with the reference direction and the reference scale.

[0151] Returning to FIG. 9, when the process of step S606 is completed, the control device 110 ends the process of this flowchart.

[0152] In this way, in this example, based on the relationship between the reference data and the comparison data organized as described above, the control device 110 can specifically display on the display device 140 a comparison of combinations of reference data and comparison data corresponding to common parts of the large structures being compared. This allows the user to compare two sets of data corresponding to common parts of the large structures being compared, and to examine changes over time in the same large structure or differences between large structures of the same design (same type).

[0153] [Other examples of testing support devices] Next, other examples of the method for supporting the comparison test will be described.

[0154] The first to fifth examples described above may be combined as appropriate.

[0155] For example, in the fourth and fifth examples described above, the test support device 100 may employ functions similar to those of the difference detection unit 1107 and the input condition determination unit 1100, as in the second and third examples described above.

[0156] Furthermore, the first to fifth examples described above may be modified or changed as appropriate.

[0157] For example, in the first to fifth examples described above, the inspection support device 100 may further assist the user in creating a report of the results of the comparison inspection. Specifically, the inspection support device 100 (control device 110) may assist the user in inputting, into a predetermined file format for the inspection result report, a combination of image data corresponding to a portion in which a difference was found among the common portions of the large structures being compared. The portion in which a difference was found may be, for example, a portion in which a difference was found among the common portions of the large structures being compared, based on the combination of image data displayed on the display device 140 in the first or third example described above. Furthermore, the portion in which a difference was found may be, for example, a portion in which a difference was detected by the difference detection unit 1107 among the common portions of the large structures being compared, in the second example described above.

[0158] Furthermore, for example, in the second example described above, the inspection support device 100 may automatically perform a comparative inspection instead of assisting a user in the comparative inspection. Specifically, the inspection support device 100 (shape estimation unit 1102) may receive image data sets transmitted from a drone operating autonomously as the sensor device 200 and automatically generate overall shape data from the image data sets of the large structures. Furthermore, the inspection support device 100 (link processing unit 1104) may automatically associate image data of both image data sets with portions of the external shapes based on the automatically generated overall shape data of both large structures to be compared, and automatically register the associations in the link DB 1105. Furthermore, the inspection support device 100 (difference detection unit 1107) may extract combinations of image data corresponding to common portions of both overall shape data corresponding to the large structures to be compared, and automatically detect differences from the extracted combinations of image data. The inspection support device 100 may then input the combination of image data corresponding to the area where the difference was detected into the file format of the inspection results, input the inspection results regarding the content and extent of the difference, and automatically output a file regarding the inspection results.

[0159] [Other examples of inspection support systems] Next, another example of the test support system 1 according to this embodiment will be described with reference to Fig. 23. The following description will focus on the differences from the above example (Fig. 1), and the description of the same or corresponding content as the above example may be simplified or omitted.

[0160] In this example, the inspection support system 1 includes an inspection support device 100, a sensor device 200, and a terminal device 300.

[0161] The terminal device 300 is used by a user who performs a comparison test. The terminal device 300 is, for example, a stationary terminal device such as a desktop computer terminal. The terminal device 300 may also be, for example, a portable (portable) terminal device (mobile terminal) such as a smartphone, a tablet terminal, or a laptop computer terminal.

[0162] The terminal device 300 includes a control device 310 , a communication device 320 , an input device 330 , and a display device 340 .

[0163] The control device 310 controls the processing operations of the terminal device 300 .

[0164] The functions of the control device 310 are realized by any hardware or any combination of hardware and software. The control device 310 is mainly configured with a computer including, for example, a CPU, a memory device, a non-volatile auxiliary storage device, and an interface device for input / output with the outside. The control device 310 realizes various functions, for example, by loading programs installed in the auxiliary storage device into the memory device and executing them on the CPU. The memory device is, for example, an SRAM or DRAM. The auxiliary storage device is, for example, an HDD, an SSD, or a flash memory. The interface device includes, for example, an external interface for connecting to a recording medium. This allows the terminal device 300 (control device 310) to, for example, retrieve various programs from a recording medium via the external interface and install them in the auxiliary storage device.

[0165] The communication device 320 communicates with an external device (for example, the test support device 100 or the sensor device 200) through a predetermined communication line. This allows the terminal device 300 to exchange data with the external device through the communication device 320. In addition, the terminal device 300 can retrieve various programs from the external device and install them in the control device 310 (auxiliary storage device).

[0166] The function of the communication device 320 may be incorporated into the control device 310 as an interface device for the control device 310.

[0167] The input device 330 accepts various inputs from a user of the test support system 1. The input device includes, for example, an operation input device such as a keyboard, a mouse, a joystick, a touch panel, a touch pad, a button, a toggle, or a lever. The input device 330 may also include, for example, a voice input device or a gesture input device that accepts voice input or gesture input from the user. A signal corresponding to the input content from the user accepted by the input device 330 is taken into the control device 310.

[0168] The display device 340 displays an information image relating to the comparison inspection under the control of the control device 310. The display device 340 is, for example, a liquid crystal display or an organic EL display.

[0169] In this example, the sensor device 200 is communicably connected to the terminal device 300 through a predetermined communication line, and the sensing data group may be acquired by the terminal device 300. In addition, the sensing data group acquired by the sensor device 200 may be acquired by the terminal device 300 via a portable storage medium.

[0170] In this example, the test support device 100 is installed in a location away from where a user performs work related to the comparison test. The test support device 100 is, for example, a server device. The server device may be a cloud server installed outside the facility where the user performs work related to the comparison test, or may be an edge server installed inside the facility or in a nearby communication facility (for example, a base station or station building).

[0171] The control device 110 of the inspection support device 100 may be communicatively connected to the terminal device 300 via the communication device 120, and may acquire (receive) a group of sensing data of large structures acquired by the sensor equipment 200 from the terminal device 300.

[0172] Furthermore, the control device 110 may take in various inputs from the user regarding support for the comparative examination, which are received by the terminal device 300 (input device 330), through the communication device 120. This allows the control device 110 to control support for the comparative examination in accordance with the user's input to the terminal device 300. Furthermore, the control device 110 can send a control command to the terminal device 300 through the communication device 120, and cause an information image regarding the comparative examination to be displayed on the display device 340.

[0173] In this way, in this example, the test support device 100 can receive input from a remote user and control support for the comparison test through the terminal device 300. Also, in this example, information about the comparison test can be provided to the remote user through the terminal device 300 (display device 340).

[0174] [Effect] Next, the operation of the examination support system 1 (examination support device 100) according to this embodiment will be described.

[0175] In this embodiment, the inspection support system 1 includes an organizing unit (e.g., a link processing unit 1104, a link processing unit 1104A, and a link processing unit 1104B). Specifically, based on the sensing data group (an example of a first sensing data group) of one large structure (an example of a first structure) to be compared and the overall shape data of the one structure, the organizing unit organizes the relationship between the sensing data group of one structure or the overall shape data of one structure and the sensing data of the other structure (an example of a second sensing data) so that common parts of the one and other large structures (an example of a second structure) can be compared.

[0176] For example, when inspecting large structures, such as large machinery like shovels or plants like steel mills, it is necessary to acquire a large amount of sensing data focused on the area to be inspected. Furthermore, it is practically impossible to obtain sensing data from the same distance and angle each time. Therefore, for example, when inspecting changes over time, even if a set of sensing data acquired last time and this time is prepared, it may be difficult to extract a combination of sensing data corresponding to the area to be inspected. As a result, the efficiency of the comparative inspection may decrease, or the comparative inspection may become impossible altogether.

[0177] In contrast, in this embodiment, the user can compare common parts of both large structures by using a group of sensing data or overall shape data of one large structure, the relationships of which have been organized, and sensing data of the other large structure. Therefore, the inspection support device 100 can support the user in inspecting the differences between the large structures to be compared.

[0178] In this embodiment, the one and other large structures to be compared may be the same large structure at different points in time.

[0179] This allows the inspection support device 100 to support the inspection of time-series changes in the same large structure.

[0180] In this embodiment, the one and the other large structures to be compared may be different structures with the same design.

[0181] This allows the inspection support device 100 to support the inspection of differences between different structures with the same design.

[0182] Furthermore, in this embodiment, the inspection support device 100 may include an estimation unit (e.g., a shape estimation unit 1102 or a shape estimation unit 1102A). Specifically, the estimation unit may estimate the three-dimensional shapes of one and the other large structures to be compared based on a group of sensing data of one large structure and sensing data of the other large structure. Then, a reorganization unit (e.g., a link processing unit 1104 or a link processing unit 1104A) may organize the relationship between the group of sensing data of one large structure and the sensing data of the other large structure based on the three-dimensional shape data (overall shape data) of the one and the other large structures to be compared estimated by the estimation unit so as to enable comparison of common parts of the one and the other large structures.

[0183] This allows the inspection support device 100 to organize the relationship between the sensing data group of one large structure being compared and the sensing data of the other large structure in a manner that allows comparison of common parts of both large structures.

[0184] Furthermore, in this embodiment, the estimation unit (shape estimation unit 1102) may estimate (reproduce) the three-dimensional shape of one large structure to be compared based on a group of sensing data of one structure, and may also estimate (reproduce) the three-dimensional shape based on a group of sensing data of the other large structure (an example of a second group of sensing data).The organization unit (link processing unit 1104) may then associate each piece of data included in the group of sensing data of one large structure with a portion of the three-dimensional shape of the one large structure.Similarly, the organization unit (link processing unit 1104) may associate each piece of data included in the group of sensing data of the other large structure with a portion of the three-dimensional shape of the other large structure.

[0185] This allows the inspection support device 100 to organize the relationship between the sensing data groups of both large structures to be compared, specifically in a manner that allows for comparison of common parts of both large structures.

[0186] In this embodiment, the estimation unit (shape estimation unit 1102A) may estimate the three-dimensional shape of one structure and the other structure to be compared when the structures are assumed to be the same structure. The organization unit (link processing unit 1104A) may then associate each piece of data included in the sensing data group of one large structure to be compared with a portion of the three-dimensional shape of the same structure, and may also associate the sensing data of the other large structure with a portion of the three-dimensional shape of the same structure.

[0187] This allows the inspection support device 100 to organize the relationship between the sensing data group of one large structure being compared and the sensing data of the other large structure in a manner that allows comparison of common parts of both large structures.

[0188] In addition, in this embodiment, the sorting unit (link processing unit 1104B) compares the overall shape data of one large structure (reference large structure) to be compared, which is prepared in advance, and a group of sensing data of one large structure whose parts in the overall shape data have been identified, with the sensing data of the other large structure (comparison large structure), and may associate the sensing data of the other large structure with parts in the three-dimensional shape of the one large structure that are common to the parts of the other large structure that correspond to the sensing data of the other large structure, or with data corresponding to those common parts included in the sensing data group of the one large structure.

[0189] This allows the inspection support device 100 to organize the relationship between the overall shape data or sensing data group of one large structure being compared and the sensing data of the other large structure in a manner that allows for comparison of common parts of both large structures.

[0190] In addition, in this embodiment, the link processing unit 1104 may associate corresponding common parts in the three-dimensional shape data (overall shape data) of one large structure and the three-dimensional shape data (overall shape data) of the other large structure.

[0191] As a result, the inspection support device 100 can use common parts of the three-dimensional shapes of both large structures to organize the relationship between the sensing data groups of the large structures being compared, specifically in a manner that allows the common parts of both large structures to be compared.

[0192] Furthermore, in this embodiment, the inspection support device 100 may include a difference detection unit 1107. Specifically, the difference detection unit 1107 may detect differences between common parts of one and the other large structures based on a group of sensing data or three-dimensional shape data (overall shape data) of one large structure to be compared, the relationships of which have been organized by the organization unit, and sensing data of the other large structure.

[0193] This allows the user to perform a comparative test using the detection results, and therefore the test support device 100 can improve the convenience for the user and the efficiency of the comparative test.

[0194] Furthermore, in this embodiment, the difference detection unit 1107 may select the parts in which differences have been detected from the common parts of both large structures being compared, according to at least one of the content and degree of the difference.

[0195] This allows the user to be provided with information on only a combination of image data of areas that are limited in at least one of the content and degree of difference from among common areas of both large structures to be compared, for example. Therefore, the inspection support device 100 can improve user convenience and also improve the efficiency of comparative inspections.

[0196] In this embodiment, the difference detection unit 1107 may also label each of the data in both sensing data groups corresponding to a common portion of both large structures being compared, to indicate a feature. Then, the difference detection unit 1107 may detect a difference between the common portion of both large structures based on the difference between the labels of the combinations of sensing data corresponding to the common portion of both large structures.

[0197] This allows the inspection support device 100 to specifically detect differences between common parts of both large structures being compared.

[0198] Furthermore, in this embodiment, the inspection support device 100 may include an input condition determination unit 1100. Specifically, the input condition determination unit 1100 may determine whether or not each of the sensing data groups of both large structures to be compared includes data of a specified portion that is common to both large structures.

[0199] This allows the test support device 100 to prevent a situation in which, for example, data on the test target area is insufficient in the sensing data group, making it impossible to perform a proper comparative test. As a result, the test support device 100 can improve the efficiency of the comparative test.

[0200] In addition, in this embodiment, at least one of the sensing data groups of the two large structures being compared may include multiple data of the same area acquired when at least one of the position and orientation of the sensor device 200 is different from each other.

[0201] This allows the inspection support device 100 to reproduce the three-dimensional shape of the large structure to be compared with higher accuracy.

[0202] Furthermore, in this embodiment, the inspection support device 100 may include a display unit (for example, a comparison display unit 1106, a comparison display unit 1106A, or a comparison display unit 1106B). Specifically, the display unit may display on the display device 140, 340 a combination of sensing data corresponding to common parts of both large structures to be compared, based on a group of sensing data of one large structure and sensing data of the other large structure, the relationships of which have been organized by the link processing unit 1104, so that the combinations of sensing data can be compared.

[0203] This allows the user to compare combinations of image data corresponding to common parts of both large structures to be compared and to relatively easily inspect for differences between them. Therefore, the inspection support device 100 can improve user convenience and the efficiency of comparative inspections.

[0204] In addition, in this embodiment, the display unit may define a common reference direction and a common reference scale for a common portion of both large structures to be compared.

[0205] This allows the inspection support device 100 to display on the display devices 140, 340 a combination of image data corresponding to a common portion of both large structures being compared, aligned to the reference direction and scale of that portion. This allows the user to compare image data aligned to the same direction and scale. Therefore, the inspection support device 100 can improve user convenience and the efficiency of comparative inspections.

[0206] In this embodiment, the reference direction may be defined as a direction seen from a common portion of both large structures being compared.

[0207] As a result, the inspection support device 100 can appropriately align the combination of image data corresponding to the common parts of both large structures to the same direction even if the postures of the common parts of both large structures defined by the movable parts are different.

[0208] In this embodiment, the display unit may also highlight and display the differences between the combinations of image data corresponding to common parts of both large structures being compared.

[0209] This allows the user to easily understand the differences between the combinations of image data, and therefore the examination support device 100 can improve user convenience and the efficiency of comparative examinations.

[0210] In this embodiment, the display unit may also cause the display devices 140 and 340 to display whole images (large structure images) of both large structures to be compared.

[0211] This allows the user to perform a comparative inspection while checking not only a combination of image data of specific regions but also the overall image of the large structure being compared. Therefore, the inspection support device 100 can improve user convenience and the efficiency of the comparative inspection.

[0212] In addition, in this embodiment, the display unit may display a combination of sensing data corresponding to a combination of common parts of both large structures being compared, in association with parts of the entire image of the large structure.

[0213] This allows the user to easily understand the location of the area corresponding to the combination of image data displayed on the display devices 140, 340 while comparing it with the area in the overall image of the large structure. Therefore, the inspection support device 100 can improve user convenience and also improve the efficiency of comparative inspections.

[0214] In addition, in this embodiment, the display unit may display information regarding the differences between common parts of both large structures, in correspondence with parts of the overall image, based on the group of sensing data of both large structures being compared, organized by the link processing unit 1104.

[0215] This allows the user to more easily understand the differences between the combinations of image data displayed on the display devices 140 and 340. Therefore, the examination support device 100 can improve the convenience for the user and also improve the efficiency of the comparison examination.

[0216] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist described in the claims.

[0217] Finally, this application claims priority based on Japanese Patent Application No. 2020-194575, filed on November 24, 2020, the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0218] 1. Inspection support system 100 Inspection support device (information processing device) 110 Control device 120 Communication equipment 130 Input Device 140 Display device 200 Sensor equipment (sensors) 300 Terminal Equipment 310 Control device 320 Communication Equipment 330 Input Device 340 Display device 1100 Input condition determination unit (determination unit) 1101 Image data storage unit 1102,1102A,1102B Shape estimation section (estimation section) 1103, 1103A Overall shape data storage unit 1104, 1104A, 1104B Link processing unit (organization unit) 1105, 1105A, 1105B Link DB 1106,1106A,1106B Comparison display section (display section) 1107 Difference detection unit (detection unit) 1108 Comparison data storage unit 1109 Reference data storage unit

Claims

1. a first sensing data group relating to the shape of a first structure, in which a correspondence relationship between a portion of the first structure corresponding to each piece of data included in the first sensing data group and a portion of the second structure corresponding to second sensing data relating to the shape of a second structure has not been established, and data on the three-dimensional shape of the first structure, the first sensing data group comprising an organizing unit that organizes a relationship between the first sensing data group or the data on the three-dimensional shape of the first structure and the second sensing data, so that common portions of the first structure and the second structure can be compared; Information processing device.

2. The first structure and the second structure are the same structure at different times, or different structures of the same design. The information processing device according to claim 1 .

3. an estimation unit that estimates three-dimensional shapes of the first structure and the second structure based on the first sensing data group and the second sensing data; the organizing unit organizes the relationship between the first sensing data group and the second sensing data based on the data of the three-dimensional shapes of the first structure and the second structure estimated by the estimation unit so as to enable comparison of common parts of the first structure and the second structure.

3. The information processing device according to claim 1 or 2.

4. the estimation unit estimates a three-dimensional shape of the first structure based on the first sensing data group, and estimates a three-dimensional shape of the second structure based on a second sensing data group that is a collection of the second sensing data; the organizing unit associates each piece of data included in the first sensing data group with a portion of the three-dimensional shape of the first structure, and associates each piece of data included in the second sensing data group with a portion of the three-dimensional shape of the second structure; The information processing device according to claim 3 .

5. the estimation unit estimates a three-dimensional shape of the same structure when the first structure and the second structure are assumed to be the same structure; the organizing unit associates each piece of data included in the first sensing data group with a portion of the three-dimensional shape of the identical structure, and associates the second sensing data with a portion of the three-dimensional shape of the identical structure; 4. The information processing device according to claim 3.

6. the organizing unit compares the first sensing data group, which is prepared in advance and includes data of the three-dimensional shape of the first structure and a specified portion in the three-dimensional shape data, with the second sensing data, and associates the second sensing data with a portion in the three-dimensional shape of the first structure that is common to a portion of the second structure corresponding to the second sensing data, or with data included in the first sensing data group that corresponds to the common portion; 3. The information processing device according to claim 1 or 2.

7. the organizing unit associates common portions of the three-dimensional shape data of the first structure and the three-dimensional shape data of the second structure with each other. The information processing device according to claim 5 .

8. a detection unit that detects differences between common portions of the first structure and the second structure based on the first sensing data group or the three-dimensional shape data of the first structure, the relationships of which have been organized by the organization unit, and the second sensing data; The information processing device according to claim 1 .

9. the detection unit selects a portion in which a difference has been detected from among the common portions of the first structure and the second structure, according to at least one of the content of the difference and the degree of the difference; The information processing device according to claim 8 .

10. the detection unit performs labeling to represent features on each of the data of the first sensing data group corresponding to a common portion of the first structure and the second structure and the second sensing data, and detects a difference between the common portion of the first structure and the second structure based on a difference in the labels of each piece of data. The information processing device according to claim 8 or 9.

11. a determination unit that determines whether or not the first sensing data group and the second sensing data group, which is a collection of the second sensing data, each include data of a predetermined portion that is common to the first structure and the second structure; The information processing device according to claim 1 .

12. At least one of the first sensing data group and the second sensing data group, which is a collection of the second sensing data, includes a plurality of data of the same part acquired in a state where at least one of the position and the orientation of the sensor is different from each other. The information processing device according to claim 1 .

13. a display unit that displays, based on the first sensing data group or the three-dimensional shape data of the first structure and the second sensing data whose relationships have been organized by the organization unit, the data of the first sensing data group and the second sensing data corresponding to a combination of common parts of the first structure and the second structure so as to be comparable; The information processing device according to any one of claims 1 to 12.

14. the display unit defines a common direction and a common scale for a common portion of the first structure and the second structure; The information processing device according to claim 13.

15. the common direction is defined as a direction seen from a common portion of the first structure and the second structure; The information processing device according to claim 14.

16. the display unit displays, in an emphasized manner, a difference between the data of the first sensing data group corresponding to a common portion of the first structure and the second structure and the second sensing data.

16. The information processing device according to claim 13.

17. the display unit displays an entire image of the first structure and the second structure.

17. The information processing device according to claim 13.

18. the display unit displays the data of the first sensing data group and the second sensing data corresponding to a combination of common parts of the first structure and the second structure in association with parts of the entire image. The information processing device according to claim 17.

19. the display unit displays information about differences between common portions of the first structure and the second structure in association with portions of the entire image based on the first sensing data group and the second sensing data whose relationships have been organized by the organization unit.

19. The information processing device according to claim 17 or 18.

20. The information processing device includes a reorganizing step of reorganizing a relationship between the first sensing data group or the data on the three-dimensional shape of the first structure and the second sensing data, based on a first sensing data group relating to the shape of a first structure, in which a correspondence relationship between a portion of the first structure corresponding to each data included in the first sensing data group and a portion of the second structure corresponding to second sensing data relating to the shape of a second structure has not been established, and data on the three-dimensional shape of the first structure, so that common portions of the first structure and the second structure can be compared. Information processing methods.

21. and causing an information processing device to execute an organizing step of organizing a relationship between a first sensing data group relating to a shape of a first structure, the first sensing data group having no established correspondence between a portion of the first structure corresponding to each piece of data included in the first sensing data group and a portion of the second structure corresponding to second sensing data relating to a shape of a second structure, and data on the three-dimensional shape of the first structure, so that common portions of the first structure and the second structure can be compared with each other. program.

Images