Inspection assistance device and inspection assistance method

Abstract

The present invention enables an operator to efficiently inspect a workpiece.　This inspection assistance device includes: a real data acquisition unit that acquires real data indicating the shape, position, and orientation of a workpiece placed in a real space; a conversion unit that, with respect to an inspection position on design data indicating a design shape of the workpiece, performs a conversion operation for superimposing the design data on the real data, thereby calculating an inspection position on the workpiece in the real space; a sensor position acquisition unit that acquires a position in the real space of a sensor for inspecting the workpiece; and a display control unit that causes a display device to display, as a direction in which to move the sensor, a direction in which the difference between the inspection position on the workpiece in the real space and the position of the sensor falls within an allowable position deviation range.

Description

Inspection support device and inspection support method

[0001] The present invention relates to an inspection support device and an inspection support method. The present invention claims priority to Japanese Patent Application No. 2023-096394, filed on June 12, 2023, and the contents of that application are incorporated by reference into the present application in designated states where incorporation by reference of documents is permitted.

[0002] When a robot equipped with an inspection sensor automatically inspects a workpiece, a method has been proposed in which the sensor is moved to a target position on the workpiece (Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-189003

[0004] However, to enable more flexible inspection, workers, rather than robots, sometimes manually inspect parts or other workpieces for defects by holding a sensor. When measuring a point of interest on a workpiece with a sensor, the worker determines the position and direction of the sensor based on experience, which makes it difficult to perform appropriate measurements efficiently.

[0005] The technology described in Patent Document 1 is a technology for guiding a robot to a target position on a workpiece, and cannot be applied to cases where inspection is performed manually by a person.

[0006] The present invention has been made in view of the above circumstances, and has as its object to enable workers to inspect workpieces efficiently.

[0007] The present application includes a number of means for solving at least part of the above problems, examples of which are as follows.

[0008] In order to solve the above problem, an inspection support device according to one embodiment of the present invention comprises: a reality data acquisition unit that acquires reality data indicating the shape, position, and posture of a workpiece placed in a real space; a conversion unit that calculates an inspection position on the workpiece in a real space by performing a conversion operation to overlay design data indicating the design shape of the workpiece on the inspection position on the design data; a sensor position acquisition unit that acquires the position in the real space of a sensor that inspects the workpiece; and a display control unit that causes a display device to display, as a direction in which the sensor should be moved, a direction in which the difference between the inspection position on the workpiece in a real space and the position of the sensor falls within an allowable positional deviation range.

[0009] According to the present invention, an operator can efficiently inspect a workpiece.

[0010] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0011] FIG. 1 is a schematic diagram showing an example of the configuration of an inspection support system according to the first embodiment. FIG. 2A is a schematic diagram showing an example of a defect detection sensor (part 1) according to the first embodiment. FIG. 2B is a schematic diagram showing an example of a defect detection sensor (part 2) according to the first embodiment. FIG. 2C is a schematic diagram showing an example of a film thickness or material measurement sensor according to the first embodiment. FIG. 2D is a schematic diagram showing an example of an internal defect detection sensor according to the first embodiment. FIG. 3 is a schematic diagram showing an example of a data structure of inspection condition information according to the first embodiment. FIG. 4 is a schematic diagram showing an example of a data structure of inspection result information according to the first embodiment. FIG. 5 is a schematic diagram showing an example of a conversion operation performed by a conversion unit according to the first embodiment. FIG. 6A is a schematic diagram showing an example of a display mode (part 1) of the display device according to the first embodiment. FIG. 6B is a schematic diagram showing an example of a display mode (part 2) of the display device according to the first embodiment. FIG. 6C is a schematic diagram showing an example of a display mode (part 3) of the display device according to the first embodiment. FIG. 6D is a schematic diagram showing an example of a display mode (part 4) of the display device according to the first embodiment. Fig. 7 is a flowchart showing an example of processing performed by the test support device according to the first embodiment. Fig. 8 is a flowchart showing an example of processing performed by the test support device according to the first embodiment. Fig. 9 is a schematic diagram showing an example of the hardware configuration of the test support device according to the first to third embodiments.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The examples are illustrative of the present invention, and for clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0013] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0014] Examples of various types of information may be described using expressions such as "table," "list," and "queue," but the various types of information may also be expressed using data structures other than these. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0015] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0016] In the embodiments, processing performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0017] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in the embodiments, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0018] 1 is a schematic diagram showing an example of the configuration of an inspection support system according to this embodiment. The inspection support system 10 is a system that supports an operator in manually inspecting a workpiece 200 such as a component.

[0019] As an example, the inspection support system 10 includes an inspection support device 100, a 3D scanner 111, a position and orientation measurement sensor 121, a sensor 131, a display device 140, and a work table 201.

[0020] The work table 201 is a table on which a workpiece 200 such as a component is placed. In this example, a plurality of markers 124 are provided on the surface of the work table 201. An inspection position 210 is set in advance on the workpiece 200, and an operator performs inspection at the inspection position 210 using a sensor 131.

[0021] The 3D scanner 111 is a three-dimensional measuring device that measures the shape, position, and orientation of the workpiece 200 in real space. By identifying the positions of each of the multiple markers 124, the 3D scanner 111 sets a coordinate system, for example, with the surface of the workpiece table 201 as the coordinate plane. This coordinate system is hereinafter referred to as a stage coordinate system. The 3D scanner 111 then outputs reality data 112 that indicates the shape, position, and orientation of the workpiece 200 based on the stage coordinate system. By using a three-dimensional measuring device such as the 3D scanner 111, it is possible to obtain reality data 112 that indicates the shape, position, and orientation of the workpiece 200.

[0022] The sensor 131 is a device for inspecting the workpiece 200. Figures 2A to 2D are schematic diagrams showing examples of the sensor 131 used in this embodiment.

[0023] 2A is a schematic diagram showing an example of an image-based defect detection sensor 131. The defect detection sensor 131 captures an image of the surface of the workpiece 200 with a camera lens 132 while illuminating the workpiece 200 with a plurality of lights 133. If the workpiece has a defect 211 such as a scratch or crack, the defect 211 can be detected as a difference in brightness on the image.

[0024] Furthermore, by switching which of the multiple lights 133 is turned on, the direction of illumination toward the defect 211 changes, and the image of the defect 211 changes. In this example, when the right light 133 is turned on and the left light 133 is turned off, an image can be obtained in which only the left side of the defect 211 is bright. On the other hand, when the left light 133 is turned on and the right light 133 is turned off, an image can be obtained in which only the right side of the defect 211 is bright. In this way, the defect detection sensor 131 measures the position, unevenness, size, etc. of the defect 211.

[0025] 2B is a schematic diagram of an example of a three-dimensional shape sensor in which the illumination 133 of the image-based defect detection sensor 131 (FIG. 2A) is replaced with pattern illumination 133'. The pattern illumination 133' projects a striped pattern onto the workpiece 200 using laser light, a liquid crystal projector, or the like. The projected pattern is deformed by the unevenness of the surface of the workpiece 200. Utilizing this, the three-dimensional shape sensor 131 captures an image of the projected pattern with a camera lens 132 and calculates the amount of deformation of the pattern to measure the three-dimensional shape of the surface of the workpiece 200 using the principle of triangulation.

[0026] 2C is a schematic diagram showing an example of a film thickness or material measurement sensor 131 using a reflection spectrum. The sensor 131 projects light from a spectrometer lens 134 onto a film 212 on the surface of the workpiece 200 and measures the spectral characteristics of the reflected light. The spectral characteristics change depending on variations in the thickness and material of the film 212. As a result, the sensor 131 can measure variations in the thickness and material of the film 212 by measuring and analyzing the spectral characteristics.

[0027] 2D is a schematic diagram of an example of an ultrasonic internal defect detection sensor 131. The internal defect detection sensor 131 has an ultrasonic flaw detector 135, which is brought into contact with the workpiece 200 to emit ultrasonic waves that are reflected by the internal defect 213 and return to the ultrasonic flaw detector 135. The internal defect 213 is detected by analyzing the signal. Using a similar principle, the thickness of the workpiece 200 can also be detected by the internal defect detection sensor 131. Furthermore, even if eddy currents are used instead of ultrasonic waves, the internal defect detection sensor 131 can also detect the internal defect 213 and the thickness of the workpiece 200.

[0028] Referring again to Figure 1, the sensor 131 is fitted with an attachment 122 having a plurality of markers 123 provided thereon.

[0029] The position and orientation measuring sensor 121 is a motion capture camera that measures the position and orientation of the sensor 131 in three-dimensional real space by recognizing markers 123 on the attachment 122. By providing multiple position and orientation measuring sensors 121, it is possible to measure the three-dimensional positions of the markers 123 by stereoscopic vision. Furthermore, in this embodiment, three or more markers 123 are provided on the attachment 122. This allows the position and orientation measuring sensor 121 to measure the three-dimensional orientation and position of the sensor 131 attached to the attachment 122.

[0030] Furthermore, the position and orientation measurement sensor 121 identifies the above-mentioned stage coordinate system, which uses the surface of the work table 201 as its coordinate plane, by recognizing the position of the marker 124 on the work table 201. The position and orientation measurement sensor 121 then outputs sensor position and orientation data 113 indicating the position and orientation of the sensor 131 based on the stage coordinate system. The position of the sensor 131 is, for example, the position of the center of gravity of the sensor 131 in the stage coordinate system. The orientation of the sensor 131 is the direction in which the sensor 131 is facing. If the sensor 131 has an optical system, the optical axis direction of the optical system is the direction in which the sensor 131 is facing. This direction may be the direction in which one of the three coordinate axes of the local coordinate system set in the sensor 131 is facing.

[0031] The position and orientation measuring sensor 121 is not limited to a motion capture camera, and other sensors may be used as the position and orientation measuring sensor 121. One such sensor is a sensor called a laser tracker. This sensor irradiates a retroreflective mirror (not shown) attached to the attachment 122 with laser light, and measures the direction and distance of the mirror based on the deviation of the reflected light and the round-trip time of the laser light. In this case, the orientation of the sensor 131 can also be measured by recognizing the positions of multiple markers 123 with a camera. Note that a single laser tracker is sufficient instead of multiple ones.

[0032] The inspection support device 100 is a computer such as a PC (Personal Computer) or a server, and is a device that supports an operator in inspecting a workpiece 200 using a sensor 131. As an example, the inspection support device 100 includes an overall control unit 104, a storage unit 105, an input unit 107, a display unit 108, a memory unit 109, a reality data acquisition unit 110, a sensor position acquisition unit 120, and a sensor data acquisition unit 130.

[0033] The storage unit 109 stores inspection condition information 101, CAD (Computer Aided Design) data 102, and inspection result information 106.

[0034] The CAD data 102 is three-dimensional CAD data of the workpiece 200, and is an example of design data that indicates the designed shape of the workpiece 200. For example, the CAD data 102 is data that collects the position coordinates of each point on the surface of the workpiece 200. Hereinafter, the coordinate system that is the reference for the position coordinates in the CAD data 102 will be referred to as the workpiece coordinate system.

[0035] 3 is a schematic diagram showing an example of the data structure of the inspection condition information 101. The inspection condition information 101 is information that associates an inspection position, an inspection method, and an allowable range with each other.

[0036] The inspection position is the position coordinate of the inspection position on the CAD data 102 that employs the aforementioned workpiece coordinate system, and is a position that is planned to be inspected in advance on the surface of the workpiece 200 indicated by the CAD data 102.

[0037] The inspection method is the ID of the sensor 131 used for inspection at the inspection position and the inspection direction (posture) of the sensor 131 during inspection. The inspection direction is set based on the workpiece coordinate system.

[0038] The inspection direction is the direction (posture) in which the sensor 131 faces during inspection, and is, for example, the extension direction of the reference line of the sensor 131. If the sensor 131 has an optical system, the reference line of the sensor 131 is, for example, the optical axis of the optical system. The inspection direction is the normal direction at the inspection position of the workpiece 200 indicated by the CAD data 102. However, depending on the type of sensor 131, the inspection direction may be a direction tilted from the normal direction.

[0039] The allowable ranges are an allowable positional deviation range, an allowable distance range, and an allowable angle range. The allowable positional deviation range is an allowable range of deviation between the inspection position in real space and the position of the sensor 131 when these positions are projected onto a plane perpendicular to the reference line of the sensor 131. The allowable distance range is an allowable range of the distance between the inspection position in real space and the position of the sensor 131. The allowable angle range is an allowable range of the angle between the inspection direction and the direction in which the sensor 131 is actually facing in real space.

[0040] 4 is a schematic diagram showing an example of the data structure of the inspection result information 106. The inspection result information is information that associates an inspection position on the CAD data 102 with the inspection result at that inspection position. The inspection result is the result of inspection at the inspection position by the sensor 131. For example, the sensor data 114 and information indicating the presence or absence of an abnormality, such as "scratch present" or "no scratch present," are stored in the inspection result.

[0041] Referring again to FIG. 1 , the reality data acquisition unit 110 acquires reality data 112 indicating the shape, position, and orientation of the workpiece 200 relative to the stage coordinate system from the 3D scanner 111. Note that in this example, the 3D scanner 111 performs the conversion to the stage coordinate system, but the reality data acquisition unit 110 may also perform the conversion. In this case, the 3D scanner 111 outputs shape data indicating the shape of the workpiece 200 and position data indicating the positions of the markers 124 to the reality data acquisition unit 110. Then, based on the shape data and position data, the reality data acquisition unit 110 generates reality data 112 indicating the shape, position, and orientation of the workpiece 200 relative to the stage coordinate system.

[0042] The sensor position acquisition unit 120 acquires sensor position and orientation data 113 indicating the position and orientation of the sensor 131 with respect to the stage coordinate system from the position and orientation measurement sensor 121. Note that in this example, the position and orientation measurement sensor 121 performs the conversion to the stage coordinate system, but the sensor position acquisition unit 120 may also perform the conversion. In this case, the position and orientation measurement sensor 121 outputs shape data indicating the shape of the sensor 131 and position data indicating the position of the marker 123 to the sensor position acquisition unit 120. Then, the sensor position acquisition unit 120 generates sensor position and orientation data 113 indicating the position and orientation of the sensor 131 with respect to the stage coordinate system based on the shape data and the position data.

[0043] The sensor data acquisition unit 130 acquires sensor data 114 indicating the inspection results of the sensor 131 .

[0044] The overall control unit 104 has a function of controlling the entire test support device 100, and includes, for example, a conversion unit 104a and a display control unit 104b.

[0045] The conversion unit 104a calculates the inspection position on the workpiece 200 in real space by performing a conversion operation on the inspection position in the inspection condition information 101, superimposing the CAD data 102 on the real data 112. Furthermore, the conversion unit 104a performs the above conversion operation on the inspection direction included in the inspection method of the inspection condition information 101, thereby converting the inspection direction into an inspection direction on the workpiece 200 in real space.

[0046] 5 is a schematic diagram showing an example of a conversion operation performed by the conversion unit 104a. In FIG. 5, a workpiece 200 in the stage coordinate system in real space is shown together with a designed workpiece 300 represented by the CAD data 102 in the workpiece coordinate system.

[0047] First, the conversion unit 104 a acquires the real data 112 from the real data acquisition unit 110 and acquires the CAD data 102 from the storage unit 109 .

[0048] The conversion unit 104a then identifies a conversion operation that causes the workpiece 300 represented by the CAD data 102 to overlap the workpiece 200 represented by the reality data 112. Such a conversion operation is a combination of a rotation operation T for aligning the orientations of the coordinate axes of the workpiece coordinate system and the stage coordinate system, and a translation operation v for translating the workpiece coordinate system so that its origin coincides with the origin of the stage coordinate system. By performing such a conversion operation on the workpiece 300, the workpiece 300 can be overlapped on the workpiece 200.

[0049] Hereinafter, the matrix representing the rotation operation T will also be referred to as the rotation matrix T. Hereinafter, the vector representing the translation operation v will also be referred to as the translation vector v.

[0050] Furthermore, the positions, shapes, and orientations of workpieces 300 and 200 are respectively represented by CAD data 102 and real-world data 112. Therefore, hereinafter, performing a conversion operation on workpiece 300 is also referred to as performing a conversion operation on CAD data 102. Furthermore, overlaying workpiece 300 on workpiece 200 is also referred to as overlaying CAD data 102 on real-world data 112.

[0051] The conversion unit 104a determines, from among such conversion operations, a rotation operation T and a translation operation v that constitute a conversion operation that minimizes the deviation between the converted CAD data 102 and the real data 112. As an algorithm for overlaying the CAD data 102 on the real data 112 in this way, an iterative distance deviation minimization algorithm called ICP (Iterative Closest Point) can be used. This makes it possible to obtain a conversion operation that minimizes the difference between the real data 112 and the converted CAD data 102.

[0052] The conversion unit 104a calculates the inspection position 210 on the workpiece 200 by applying the conversion operation thus obtained to the inspection position 310 on the workpiece 300. For example, if the position vector of the inspection position 310 in the workpiece coordinate system is Ap, the conversion unit 104a calculates the position vector of the inspection position 210 in the stage coordinate system as T·Ap+v.

[0053] Furthermore, the conversion unit 104a converts the inspection direction Bp on the workpiece 300 into an inspection direction T·Bp on the workpiece 200 by a rotation operation T that constitutes the above conversion operation.

[0054] The position vector Ap is stored in advance as the inspection position in the inspection condition information 101 (see FIG. 3). The inspection direction Bp is stored in advance as the inspection method in the inspection condition information 101. If the inspection direction Bp is not defined in the inspection method, the normal direction of the position vector Ap of the workpiece 300 may be used as the inspection direction Bp.

[0055] By the above operations, the inspection support device 100 can identify the inspection position T·Ap+v and the inspection direction T·Bp on the workpiece 200 in the real space.

[0056] 1 again, the display control unit 104b displays on the display device 140 the movement direction of the sensor 131 that reduces the positional deviation between the inspection position and the sensor 131, the deviation between the inspection direction and the attitude of the sensor 131, and the like.

[0057] 6A to 6D are schematic diagrams showing examples of display modes of the display device 140. FIG.

[0058] 6A shows an example in which a tablet terminal is used as the display device 140. An image showing the movement direction 141, swing direction 142, and rotation direction 144 of the sensor 131 is displayed on the display device 140. In this example, an arrow is displayed as the image.

[0059] Movement method 141 is a direction in which the amount of deviation between the inspection position T·Ap+v on the workpiece 200 and the actual position of the sensor 131 when projected onto a plane perpendicular to the direction in which the sensor 131 is facing falls within the allowable position deviation range included in the “allowable range” of the inspection condition information 101.

[0060] By moving the sensor 131 in the movement direction 141, the worker can keep the positional deviation of the sensor 131 within the allowable positional deviation range without relying on his or her own experience.

[0061] The swing direction 142 is a direction in which the sensor 131 is swung while one point of the sensor 131 is fixed, and the angle formed by the inspection direction T·Bp on the workpiece 200 and the direction in which the sensor 131 is actually facing is within the allowable angle range included in the "allowable range" of the inspection condition information 101. The example of Fig. 6A shows a "pan" in which the sensor 131 is swung from below to above.

[0062] By changing the attitude of the sensor 131 in accordance with the swing direction 142, the worker can keep the deviation of the attitude of the sensor 131 within the allowable angle range without relying on his or her own experience.

[0063] The rotation direction 144 indicates the direction in which the sensor 131 is rotated around the reference line of the sensor 131 as the rotation axis. For example, the rotation direction 144 indicates the direction in which the angle between the direction perpendicular to the reference line of the sensor 131 and the reference direction set at the inspection position T·Ap+v on the workpiece 200 (for example, the direction in the tangent plane of the workpiece 200 at that inspection position) falls within an allowable range. The value of the allowable range is stored, for example, in the allowable range of the inspection condition information 101.

[0064] The display device 140 also displays a distance direction 143. The distance direction 143 indicates the direction in which the distance between the inspection position T·Ap+v on the workpiece 200 and the position of the sensor 131 falls within the allowable distance range included in the "allowable range" of the inspection condition information 101. The distance direction 143 has an inner circle and an outer circle. The inner circle indicates the allowable distance range included in the allowable range of the inspection condition information 101 (see FIG. 3), and its size is fixed. On the other hand, the size of the outer circle changes depending on the distance between the inspection position T·Ap+v and the actual position of the sensor 131. When the distance falls within the allowable distance range, the outer circle falls within the inner circle. On the other hand, when the distance exceeds the allowable distance range, the outer circle becomes larger than the inner circle.

[0065] By moving the sensor 131 closer to or farther away from the workpiece 200 so that the outer circle of the distance direction 143 fits within the inner circle, the worker can bring the distance between the inspection position T·Ap+v and the actual position of the sensor 131 within the allowable distance range without relying on his or her own experience.

[0066] The display control unit 104b identifies the deviations between the actual position and orientation of the sensor 131 indicated by the sensor position and orientation data 113 and the inspection position T·Ap+v and inspection direction T·Bp on the workpiece 200, and displays each direction 141, 142, 144 and distance direction 143 depending on whether each deviation falls within the allowable position deviation range, allowable distance range, and allowable angle range, respectively.

[0067] Furthermore, the display control unit 104b may change the display mode of the display device 140 when the deviation falls within an allowable range. For example, when the difference between the inspection position and the actual position of the sensor 131 falls within the allowable position deviation range, the arrows of the movement direction 141 and the rotation direction 144 may be erased or made lighter. This allows the operator to recognize that there is no need to move the sensor 131 any further and that inspection can be started with the sensor 131.

[0068] Furthermore, the display control unit 104b may display the magnitude of the deviation by the color, density, size, blinking speed, etc. of the directions 141 to 144.

[0069] Furthermore, a small display may be used as the display device 140 and attached to the attachment 122 .

[0070] 6B shows an example in which an eyeglass-type display device (e.g., smart glasses) is used as the display device 140. By using the eyeglass-type display device, directions 141 to 144 are displayed in the worker's field of vision, allowing the worker to work efficiently.

[0071] FIG. 6C shows an example of a CG (Computer Graphics) image of the workpiece 200 displayed on the display device 140. This CG image is an image of the workpiece 200 as seen from the sensor 131, and the inspection position 210 is also displayed in the CG image. For example, the display control unit 104b generates a CG image including the inspection position 210 based on the CAD data 102 that has been converted as shown in FIG. 5. The display control unit 104b may also generate a CG image based on the reality data 112. Furthermore, the display control unit 104b generates a CG image as seen from the sensor 131 by determining the relative positional relationship between the workpiece 200 and the sensor 131 in real space based on the sensor position and orientation data 113 and the reality data 112. The display control unit 104b also displays a crosshair 145 indicating the position of the sensor 131 on the display device 140. The deviation between the intersection of the crosshair 145 and the inspection position 210 is the positional deviation. Therefore, in this example, it is unnecessary to display the movement direction 141 and the rotation direction 144, which indicate the positional deviation.

[0072] The swing direction 142 may be indicated by making each of the inner and outer circles of the distance direction 143 elliptical with the swing direction 142 as its minor axis. In this case, the inner and outer circles may be made perfect circles when the angle between the inspection direction and the direction in which the sensor 131 is facing falls within the allowable angle range. This makes it unnecessary to display the swing direction 142.

[0073] 6D shows an example in which a display such as a light-emitting diode or a liquid crystal display is provided on the attachment 122 as the display device 140. In FIG. 6D, the black parts indicate the light-emitting parts. In this way, by incorporating an arrow or circle light-emitting pattern into the display device 140 in advance and illuminating the light-emitting pattern when the deviation exceeds the tolerance, the worker can grasp the direction in which the deviation will fall within the tolerance. Furthermore, the magnitude of the deviation may be expressed by the color, intensity, size, blinking speed, etc. of the light-emitting pattern.

[0074] Referring back to FIG. 1 , the storage unit 105 generates inspection result information 106 by associating the inspection positions on the CAD data 102 with the inspection results at those inspection positions, and stores the inspection result information 106 in the memory unit 109. For example, the storage unit 105 acquires the inspection positions on the CAD data 102 via the conversion unit 104a, and acquires sensor data 114 including the inspection results at those inspection positions from the sensor data acquisition unit 130. The storage unit 105 then associates the inspection positions with the inspection results and stores them in the inspection result information 106. This allows the operator to later verify where on the CAD data 102 an abnormality exists.

[0075] The input unit 107 is an input device such as a keyboard or a mouse that allows an operator to input various data to the inspection support device 100 .

[0076] The display unit 108 is a display device such as a liquid crystal display that displays various data to the worker when the data is being input.

[0077] Next, the processing performed by the test support device 100 according to this embodiment will be described.

[0078] FIG. 7 is a flowchart showing an example of processing performed by the test support device 100 according to this embodiment.

[0079] First, the reality data acquisition unit 110 acquires reality data 112 indicating the shape, position, and orientation of the workpiece 200 based on the stage coordinate system from the 3D scanner 111 (step S1).

[0080] For example, the 3D scanner 111 measures the shape of the workpiece 200 on the workpiece table 201, and then identifies the positions of multiple markers 124 on the workpiece table 201 to generate the reality data 112. Note that if the entire shape of the workpiece 200 cannot be measured by measurement from one direction, the worker may move the workpiece 200 so that the 3D scanner 111 measures the shape from multiple directions.

[0081] Next, the conversion unit 104a acquires the CAD data 102 from the storage unit 109 (step S2).

[0082] Next, the transformation unit 104a calculates a rotation matrix T and a translation vector v that realize the transformation operation of superimposing the CAD data 102 on the real data 112, as shown in FIG. 5 (step S3).

[0083] Next, the display control unit 104b displays on the display device 140 a message prompting the preparation of the i-th sensor 131 (step S4). In this example, there are multiple sensors 131 identified by IDs of integers i (=1, ..., N), and when step S4 is performed for the first time, a message prompting the preparation of the sensor 131 with i=1 is displayed on the display device 140.

[0084] Based on this message, the worker turns on the power of the sensor 131 and attaches the attachment 122 to the sensor 131. Note that if the attachment 122 is already provided for each of the multiple sensors 131, there is no need to attach the attachment 122. Furthermore, depending on the type of sensor 131, the worker may perform a calibration operation for the sensor 131.

[0085] Next, the overall control unit 104 reads out the inspection condition information 101 from the storage unit 109 (step S5).

[0086] Next, the overall control unit 104 reads the pth record from the top of the inspection condition information 101 and determines whether the "inspection method" of that record specifies the use of the i-th sensor (step S6). When executing step S6 for the first time, the overall control unit 104 reads the record p=1. Even when executing step S6 for the first time, the overall control unit 104 also reads the record p=1 when switching to the next sensor 131 ("NO" in step S15). If the determination in step S6 is YES, the process proceeds to step S7, and if the determination is NO, the process proceeds to step S14.

[0087] In step S7, the display control unit 104b displays a CG image of the workpiece 200 and the inspection position 210 on the display device 140 as shown in Fig. 6C. For example, the display control unit 104b displays the position T·Ap+v obtained by performing the transformations of the rotation matrix T and the translation vector v on the inspection position Ap stored in the pth record of the inspection condition information 101 as the inspection position 210. This allows the operator to move the sensor 131 closer to the inspection position 210. Note that if a CG image is not displayed on the display device 140 as in Figs. 6A, 6B, and 6D, step S7 may be skipped.

[0088] Next, the sensor position acquisition unit 120 acquires the sensor position and orientation data 113 of the i-th sensor 131 from the position and orientation measurement sensor 121 (step S8).

[0089] Next, the display control unit 104b identifies the deviations between the actual position and orientation of the sensor 131 indicated by the sensor position and orientation data 113 and the inspection position T·Ap+v and inspection direction T·Bp on the workpiece 200. Then, the display control unit 104b displays each direction 141 to 144 on the display device 140 as shown in Figures 6A to 6D depending on whether each deviation falls within the allowable position deviation range, allowable distance range, and allowable angle range (step S9). Note that the inspection position Ap and inspection direction Bp used in step S9 are the inspection position and inspection direction in the workpiece coordinate system stored in the p-th record of the inspection condition information 101, respectively.

[0090] Next, the operator moves the sensor 131 according to the display, and the sensor position acquisition unit 120 acquires the sensor position and orientation data 113 of the sensor 131 after the movement (step S10). The display control unit 104b then determines whether the deviation is within the allowable range (step S11). If the determination here is NO, the process returns to step S7.

[0091] On the other hand, if the answer is YES, the process proceeds to step S12, where the operator performs an inspection at the pth inspection position using the i-th sensor 131, and the sensor data acquisition unit 130 acquires sensor data 114 indicating the inspection results.

[0092] Next, the storage unit 105 associates the p-th inspection position on the CAD data 102 with the inspection result at that inspection position and stores them in the inspection result information 106 (step S13).

[0093] Next, the overall control unit 104 determines whether inspection has been completed at all inspection positions (step S14). For example, the overall control unit 104 determines whether inspection has been completed at the inspection position of the last row where p is the maximum in the inspection condition information 101. If the determination here is NO, p is incremented by 1 and the process returns to step S5.

[0094] On the other hand, if the determination in step S14 is YES, the process proceeds to step S15. In step S15, the overall control unit 104 determines whether inspection has been performed on all of the first to Nth sensors 131 (step S15). For example, if i<N, the overall control unit 104 determines that inspection has not been performed on all of the sensors 131 (NO), and if i=N, the overall control unit 104 determines that inspection has been performed on all of the sensors 131 (YES).

[0095] If the determination here is NO, i is incremented by 1 and the process returns to step S4, whereas if the determination is YES, the process ends.

[0096] According to the present embodiment described above, in step S9, the display control unit 104b displays the directions 141 to 144 as shown in Figures 6A to 6D on the display device 140. This allows the operator to move the sensor 131 while relying on the directions 141 to 144, thereby keeping deviations in the position and orientation of the sensor 131 within an acceptable range. As a result, the operator can determine the position and orientation of the sensor 131 without relying on his or her own experience, and can efficiently inspect the workpiece 200.

[0097] Second Embodiment In the first embodiment, the 3D scanner 111 measures the shape, position, and orientation of the workpiece 200 to generate reality data 112. In contrast, in the present embodiment, an operator scans the workpiece 200 with the three-dimensional shape sensor 131 described in FIG. 2B , and the sensor position acquisition unit 120 measures the shape, position, and orientation of the workpiece 200 at the time of scanning. Then, the inspection support device 100 generates reality data 112 based on the sensor position and orientation data 113 generated by the sensor position acquisition unit 120. In this case, the markers 124 on the workpiece table 201 may be omitted.

[0098] As another example, a camera that captures images of the workpiece 200 may be used as the sensor 131, and the reality data 112 may be generated from the obtained image sequence using an algorithm called SFM (Structure from Motion) or visual SLAM (Simultaneous Localization and Mapping). In this case, the marker 124 is used to calibrate the magnification of the measurement results.

[0099] Third Embodiment In the first embodiment, the inspection position of the workpiece 200 is defined in advance in the inspection condition information 101 (see FIG. 3). In contrast, in the present embodiment, a new inspection position is added in addition to the already defined inspection positions.

[0100] FIG. 8 is a flowchart showing an example of processing performed by the test support device 100 according to this embodiment.

[0101] First, if the answer to step S15 in FIG. 7 is YES, the display control unit 104b identifies a position on the CAD data 102 where the difference between the CAD data 102 after the transformation T·Ap+v and the real data 112, i.e., the difference in shape, is greater than or equal to a predetermined value (step S21).

[0102] Next, the display control unit 104b identifies the position on the CAD data 102 where the abnormality was detected in the inspection in step S12 (step S22). For example, the display control unit 104b identifies the position where the abnormality was detected by referring to the inspection result information 106.

[0103] Next, the storage unit 105 updates the inspection condition information 101 by adding a record to the inspection condition information 101 that includes the positions identified in steps S21 and S22 as inspection positions (step S23). The inspection method and tolerance range for the newly added record may be set appropriately by an operator. Alternatively, the overall control unit 104 may set the inspection method and tolerance range for the newly added record. For example, the overall control unit 104 may set a predetermined inspection method and tolerance range depending on the position and shape of the location identified in step S21. Alternatively, the overall control unit 104 may set a predetermined inspection method and tolerance range depending on the position, shape, type, etc. of the location where the abnormality is present. Furthermore, the overall control unit 104 may set a sensor 131 different from that used in the previous inspection as the inspection method for the inspection position where the abnormality was detected.

[0104] Then, the process returns to step S4. This causes the process to proceed to step S12, where an additional test is performed. Different sensors 131 may be used for the additional test and the initial test performed before steps S22 and S23. For example, in the initial test, "sensor A" is assigned as the sensor 131 at i=1, and "sensor B" is assigned as the sensor 131 at i=2. In addition, in the additional test, "sensor A" is assigned as the sensor 131 at i=3, and "sensor C" is assigned as the sensor 131 at i=4. In the initial test, when i=2 is reached in step S15, a YES determination is made, and then i is set to 3 before returning to step S4. Alternatively, after completing step S23, p may be reset to 1 to perform tests using "sensor A" and "sensor C" at all test positions, or p may be reset to the value of p corresponding to the record added in step S23 before returning to step S4.

[0105] This completes the processing performed by the test support device 100 according to this embodiment. While both steps S21 and S22 were executed in the above example, only one of these steps may be executed. Furthermore, while steps S21 and S22 were executed after step S15 in this example, steps S21 and S22 may be executed at a different timing.

[0106] According to the present embodiment described above, among the positions on the CAD data 102, a position where the difference from the actual data 112 is equal to or greater than a predetermined value or a position where an abnormality is found is newly set as an inspection position. Therefore, an operator can inspect the newly set position using another sensor 131 to check what kind of abnormality exists at that position.

[0107] Next, the hardware configuration of the test support device 100 according to the first to third embodiments will be described.

[0108] FIG. 9 is a schematic diagram showing an example of the hardware configuration of the test support device 100 according to the first to third embodiments.

[0109] 9, the test support device 100 includes a storage device 100a, a memory 100b, a processor 100c, an input device 100d, a display device 100e, a communication interface 100f, and a medium reading device 100g. These components are interconnected by a bus 100i.

[0110] The storage device 100a is a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores the inspection support program 50 according to the first to third embodiments.

[0111] The inspection assistance program 50 may be recorded on a computer-readable recording medium 100h, and the processor 100c may read the inspection assistance program 50 from the recording medium 100h.

[0112] The recording medium 100h may be a physically portable recording medium such as a CD-ROM (Compact Disc - Read Only Memory), a DVD (Digital Versatile Disc), or a USB (Universal Serial Bus) memory. A semiconductor memory such as a flash memory or a hard disk drive may also be used as the recording medium 100h.

[0113] The inspection assistance program 50 may be stored in a device connected to a public line, the Internet, a LAN (Local Area Network), etc. In this case, the processor 100c may read and execute the inspection assistance program 50.

[0114] The memory 100b is hardware that temporarily stores data, such as a dynamic random access memory (DRAM), and the inspection support program 50 is developed on the memory 100b.

[0115] The processor 100c is a CPU (Central Processing Unit) or GPU (Graphical Processing Unit) that controls each part of the test support device 100. The processor 100c executes the test support program 50 in cooperation with the memory 100b, thereby realizing the overall control unit 104, storage unit 105, reality data acquisition unit 110, sensor position acquisition unit 120, and sensor data acquisition unit 130 shown in FIG.

[0116] The storage unit 109 in FIG. 1 is realized by the storage device 100a and the memory 100b.

[0117] The input device 100d is an input device such as a keyboard or a mouse for realizing the input unit 107 in FIG.

[0118] The display device 100e is a display device such as a liquid crystal display for realizing the display unit 108 in FIG.

[0119] The communication interface 100f is hardware such as a network interface card (NIC) for connecting the inspection support device 100 to a network such as the Internet or a LAN. The reality data acquisition unit 110, the sensor position acquisition unit 120, and the sensor data acquisition unit 130 in Fig. 1 acquire reality data 112, the sensor position and orientation data 113, and the sensor data 114 via the communication interface 100f.

[0120] The medium reading device 100g is hardware such as a USB reader for reading data recorded on the recording medium 100h.

[0121] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0122] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described components. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0123] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the program, decision table, and files that implement each function can be stored in memory, a storage device such as an HDD or SSD, or a recording medium such as an IC (Integrated Circuit) card, an SD (Secure Digital) card, or a DVD (Digital Versatile Disc). Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0124] 10...inspection support system, 50...inspection support program, 100...inspection support device, 100a...storage device, 100b...memory, 100c...processor, 100d...input device, 100e...display device, 100f...communication interface, 100g...medium reading device, 100h...recording medium, 100i...bus, 101...inspection condition information, 102...CAD data, 104...overall control unit, 104a...conversion unit, 104b...display control unit, 105...storage unit, 106...inspection result information, 107...input unit, 108...display unit, 109...memory unit, 110...real data acquisition unit, 111...3D scanner, 112...real data , 113...sensor position and orientation data, 114...sensor data, 120...sensor position acquisition unit, 121...position and orientation measurement sensor, 122...attachment, 123, 124...marker, 130...sensor data acquisition unit, 131...sensor, 132...camera lens, 133...lighting, 133'...pattern lighting, 134...spectroscope lens, 135...ultrasonic flaw detector, 140...display device, 141...movement direction, 142...swing direction, 143...distance direction, 144...rotation direction, 145...crosshair, 200, 300...work, 201...work table, 210, 310...inspection position, 211...defect, 212...film, 213...internal defect.

Claims

1. An inspection support device having: a reality data acquisition unit that acquires reality data indicating the shape, position, and posture of a workpiece placed in real space; a conversion unit that calculates an inspection position on the workpiece in real space by performing a conversion operation on inspection position on design data indicating the designed shape of the workpiece by overlaying the design data on the reality data; a sensor position acquisition unit that acquires the position in real space of a sensor that inspects the workpiece; and a display control unit that causes a display device to display, as a direction in which the sensor should be moved, a direction in which the difference between the inspection position on the workpiece in real space and the position of the sensor falls within an allowable position deviation range.

2. An inspection support device as described in claim 1, wherein the conversion unit converts the inspection direction on the design data into an inspection direction on the workpiece in real space by the conversion operation, the sensor position acquisition unit acquires the direction in which the sensor is facing, and the display control unit causes the display device to display, as the direction in which to swing the sensor, a direction in which the angle between the inspection direction on the workpiece in real space and the direction in which the sensor is facing, falls within an allowable angle range.

3. An inspection support device according to claim 1, further comprising: a storage unit which associates the inspection position on the design data with the inspection result at that inspection position and stores the association in a memory unit.

4. An inspection support device as claimed in claim 1, wherein the reality data acquisition unit acquires the reality data from a three-dimensional measuring device that measures the shape, position and orientation of the workpiece in real space.

5. An inspection support device as described in claim 1, wherein the conversion unit performs a rotation operation and a translation operation on the design data as the conversion operation, and performs the conversion operation on the inspection position such that the deviation between the converted design data and the actual data is minimized.

6. An inspection support device according to claim 1, wherein the sensor is any one of a defect detection sensor using an image, a film thickness or material measurement sensor using a reflection spectrum, and an internal defect detection sensor using ultrasonic waves or eddy currents.

7. An inspection support device as described in claim 1, wherein the display control unit causes the display device to display a direction in which the distance between the inspection position on the work in real space and the position of the sensor falls within an allowable distance range as a direction in which the sensor is moved closer to or farther away from the inspection position.

8. An inspection support device as described in claim 1, wherein the display control unit changes the display mode of the display device when the difference between the inspection position on the work in real space and the position of the sensor falls within the allowable position deviation range.

9. An inspection support device as described in claim 1, wherein the display control unit sets, among the positions on the design data, a position where the difference between the design data on which the conversion operation has been performed and the actual data is equal to or greater than a predetermined value as the inspection position on the design data.

10. An inspection support device according to claim 1, wherein the display control unit sets a position on the design data where an abnormality has been detected by the sensor as the inspection position on the design data.

11. An inspection support method in which a computer executes the steps of: acquiring reality data indicating the shape, position, and posture of a workpiece placed in real space; calculating an inspection position on the workpiece in real space by performing a conversion operation on inspection position on design data indicating the designed shape of the workpiece by overlaying the design data on the real data; acquiring the position in real space of a sensor that inspects the workpiece; and displaying on a display device a direction in which the difference between the inspection position on the workpiece in real space and the position of the sensor falls within an allowable positional deviation range as the direction in which the sensor will be moved.

12. An inspection support method as described in claim 11, wherein the computer further executes the steps of: converting the inspection direction on the design data into an inspection direction on the work in real space by the conversion operation; acquiring the direction in which the sensor is facing; and displaying on the display device, as the direction in which to swing the sensor, a direction in which the angle between the inspection direction on the work in real space and the direction in which the sensor is facing falls within an allowable angle range.

13. An inspection support method according to claim 11, wherein the computer further executes a step of associating the inspection position on the design data with the inspection result at that inspection position and storing them in a memory unit.

14. An inspection support method according to claim 11, wherein the computer acquires the real-world data from a three-dimensional measuring device that measures the shape, position, and orientation of the workpiece in real space.

15. An inspection support method as set forth in claim 11, wherein the computer performs a rotation operation and a translation operation on the design data as the transformation operation, and performs the transformation operation on the inspection position such that the deviation between the transformed design data and the actual data is minimized.

Images