Information processing device, information processing system, program, and information processing method

Abstract

To enable detection of defects on the work surface without using a CMOS camera. [Solution] The information processing device of the present application is a device that executes a process of accumulating the signed polarity values ​​or absolute values ​​of the polarity values ​​of event data obtained by imaging a workpiece with an EVS camera for each pixel that images the same location on the workpiece, and a process of generating a two-dimensional histogram based on the numerical value obtained by the accumulation. The information processing system of the present application includes an EVS camera that images a workpiece and the information processing device of the present application. The information processing method of the present application is a method that executes a process of accumulating the signed polarity values ​​or absolute values ​​of the polarity values ​​of event data obtained by imaging a workpiece with an EVS camera for each pixel that images the same location on the workpiece, and a process of generating a two-dimensional histogram based on the numerical value obtained by the accumulation.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, a program, and an information processing method used for detecting defects on the surface of a workpiece. [Background technology]

[0002] Conventionally, a system using a CMOS camera is known as a device for detecting defects on the surface of a workpiece (Patent Document 1). In this type of system, a light is irradiated onto the workpiece, the reflected light is captured by the CMOS camera, and the presence or absence of defects is determined based on the obtained image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-173251 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional systems using CMOS cameras, the quality of the captured image is greatly affected by the illumination angle and the camera placement angle, which can result in certain defects not being captured properly.

[0005] Furthermore, because the optimal lighting and imaging conditions differ depending on the paint color of the workpiece surface, detailed setting changes were required for each color, which was an operational burden.In addition, differences in paint color significantly change the amount of light incident on the CMOS camera, as well as the color tone, brightness, and contrast in the image, which creates the problem of defects not being properly recorded in the image due to phenomena caused by dynamic range such as whiteout and blackout.

[0006] Furthermore, some paint colors reflect light poorly, making it difficult to visually identify defects in those paint colors, which limits the detection capabilities of CMOS cameras. The same issue exists when the workpiece is transparent.

[0007] The present invention has been made in consideration of the above circumstances, and its problem to be solved is to provide a new information processing device, information processing system, program, and information processing method that can detect defects on the surface of a workpiece without using a CMOS camera. [Means for solving the problem]

[0008] [Information processing device] The information processing device of the present application is a device that executes a process of accumulating the signed polarity values ​​or absolute values ​​of polarity values ​​of event data obtained by imaging a workpiece with an EVS camera for each pixel that images the same location on the workpiece, and a process of generating a two-dimensional histogram based on the numerical values ​​obtained by the accumulation.

[0009] The information processing device may be configured to execute a process of generating a static visualized image based on the generated two-dimensional histogram. In this case, the static visualized image may be a color image or a monochrome image.

[0010] [Information Processing Systems] The information processing system of the present application is a system equipped with an EVS camera that captures an image of a workpiece and an information processing device that processes event data obtained by the EVS camera, and includes the information processing device of the present application as the information processing device.

[0011] [program] The program of the present application causes a computer to execute a process of accumulating the signed polarity values ​​or absolute values ​​of the polarity values ​​of event data obtained by imaging the workpiece with an EVS camera for each pixel that images the same location on the workpiece, and a process of generating a two-dimensional histogram based on the numerical values ​​obtained by the accumulation.

[0012] [Information processing method] The information processing method of the present application is a method in which an information processing device executes a process of accumulating the signed polarity values ​​or absolute values ​​of the polarity values ​​of event data obtained by imaging a workpiece with an EVS camera for each pixel that images the same location on the workpiece, and a process of generating a two-dimensional histogram based on the numerical values ​​obtained by the accumulation.

[0013] In this application, a "two-dimensional histogram" represents a frequency distribution as a two-dimensional lattice structure by dividing the range of values ​​of two types of variables into predetermined intervals (bins) and aggregating the frequency of occurrence for each combination of bins.

[0014] The "two-dimensional histogram" of this application includes not only a two-dimensional display (2D display) that shows the frequency of occurrence on a plane using color shading or the like, but also a three-dimensional display (3D display) that shows the frequency of occurrence in a three-dimensional manner in the vertical direction. [Effects of the Invention]

[0015] The information processing device, information processing system, program, and information processing method of the present application do not use a CMOS camera, and therefore can fundamentally solve the problems associated with conventional systems that use a CMOS camera. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a schematic diagram illustrating an example of an information processing system according to the present application. [Figure 2] FIG. 4 is an explanatory diagram showing an example of a light-transmitting portion of a lighting fixture. [Figure 3] FIG. 10 is an explanatory diagram showing an example of a two-dimensional histogram. [Figure 4] 10(a) to 10(c) are explanatory diagrams of pixels capturing an image of the same location on a workpiece. [Figure 5] (a) shows an example of a static visualization image of a workpiece without defects, and (b) shows an example of a static visualization image of a workpiece with defects. [Figure 6](a) is a graph of the cumulative polarity values ​​of events that occurred on line L1 in Figure 5(a), and (b) is a graph of the cumulative polarity values ​​of events that occurred on line L2 in Figure 5(b). [Figure 7] (a) is an explanatory diagram showing an example of the positional relationship between the EVS camera, lighting equipment, and workpiece when using reflected light, and (b) is an explanatory diagram showing an example of the positional relationship between the EVS camera, lighting equipment, and workpiece when using transmitted light. DETAILED DESCRIPTION OF THE INVENTION

[0017] (Embodiment) An example of an embodiment of the present invention will be described with reference to the drawings. The information processing device, information processing system, program, and information processing method of the present application are used to detect defects on the surface of a workpiece W. Defects include scratches, irregularities, paint peeling, color unevenness, paint unevenness, corrosion, discoloration, pinholes, cracks, foreign matter adhesion, peeling, etc.

[0018] As an example, the information processing system shown in Figure 1 includes a transport body 10 that transports a workpiece W, a lighting device 20 that illuminates the workpiece W transported by the transport body 10, an EVS camera 30 that captures images of the workpiece W transported by the transport body 10, and an information processing device 40 that processes event data acquired by the EVS camera 30.

[0019] The transport body 10 is a device that transports the workpiece W to be inspected at a predetermined speed. An existing belt conveyor, roller conveyor, or the like can be used as the transport body 10. There is no limit to the transport speed of the transport body 10, but it can be, for example, about 50 to 1000 mm / sec. However, this value is an example, and other speeds are not excluded.

[0020] The lighting device 20 is a device that illuminates a predetermined position on the transport body 10. The lighting device 20 is installed so as to be able to illuminate the entire width direction of the workpiece W passing through the imaging range. The lighting device 20 in this embodiment is ring-shaped and is arranged so as to surround the transport body 10 in a direction intersecting the transport direction of the transport body 10.

[0021] The lighting device 20 of this embodiment includes a ring-shaped housing and a light source disposed within the housing. A light-transmitting portion 21 is provided on the housing toward the light source in the emission direction, allowing light emitted from the light source to pass through.

[0022] 2, the light-transmitting portion 21 is masked in stripes. Specifically, long, thin strip-shaped masking materials 21a are attached to the light-transmitting portion 21 at predetermined intervals, so that areas irradiated with light that has passed through the portions where the masking materials 21a are not attached become bright areas, and areas irradiated with light that has passed through the portions where the masking materials 21a are attached become dark areas.

[0023] As in this embodiment, masking in a striped pattern creates light and dark areas on the surface of the workpiece W, improving the event detection accuracy of the EVS camera 30. The width of the masking material 21a can be the same as or different from the unmasked portion. Also, the translucent portion 21 can be made striped by a method other than using the masking material 21a.

[0024] The EVS camera 30 is a camera that captures an image of the workpiece W being transported by the transport body 10. The EVS camera 30 is a camera equipped with an event-based vision sensor, and outputs event data when there is a change in brightness (event) for each pixel. The event data includes information on "coordinates (x, y)," "time (timestamp)," and "polarity."

[0025] "Coordinates (x, y)" is information that identifies the pixel where the event occurred, "timestamp" is information that identifies the time when the event occurred (for example, in microseconds), and "polarity" is information that identifies the direction of the brightness change.

[0026] In this embodiment, when the "polarity" becomes brighter (when the brightness increases), a value of "+1" is defined, and when the "polarity" becomes darker (when the brightness decreases), a value of "-1" is defined. The method of defining polarity shown here is an example, and the definition of polarity may be other than "+1" and "-1", such as "1" and "2".

[0027] Event data has a data structure such as (x, y, timestamp, polarity) and is output as information such as (120, 90, 1, +1). The example shown here means that the pixel identified by x=120, y=90 became bright at the time of 1 microsecond.

[0028] There are no limitations on the specifications of the EVS camera 30 to be used, and the specifications can be selected appropriately depending on the characteristics of the workpiece W, the required detection accuracy, etc. As an example, a camera with a pixel array resolution of 1280 × 720, 640 × 480, 240 × 180, etc. can be used.

[0029] The information processing device 40 is a device that processes event data obtained by capturing images with the EVS camera 30, and can be configured as a computer (such as a PC) equipped with a processor, memory, storage, input / output unit, communication unit, etc.

[0030] The information processing device 40 generates a two-dimensional histogram based on the event data acquired by the EVS camera 30. As an example of a two-dimensional histogram, FIG. 3 shows a three-dimensional display (3D display) of the occurrence frequency in the height direction. From this two-dimensional histogram, the distribution of the occurrence frequency can be visually grasped, and defective locations can be identified.

[0031] After generating the two-dimensional histogram, the information processing device 40 generates a static visualization image that allows the defective portion to be recognized based on the generated two-dimensional histogram. The "static visualization image" here means a still image that is displayed in a visible state. The generated static visualization image may be a color image or a monochrome image.

[0032] In this embodiment, a processor of the information processing device 40 loads a program stored in a storage into a memory and executes the program to generate a two-dimensional histogram and a static visualized image based on the two-dimensional histogram. Details of the processing executed by the information processing device 40 will be described below.

[0033] The information processing device 40 executes a process (reception process) to receive event data obtained by capturing an image of the workpiece W with the EVS camera 30. In this process, the information processing device 40 receives all event data acquired by the EVS camera 30 within a predetermined capturing time.

[0034] When the event data reception process is completed, the information processing device 40 executes a process of accumulating the signed polarity value of the received event data for each pixel that captures the same location of the workpiece W (accumulation process).

[0035] Here, the "pixels capturing an image of the same location on the workpiece W" change as time t passes. For example, as shown in Figures 4(a) to 4(c), if a specific location on the workpiece W is designated as "location A," the pixels capturing the image of location A at time t=1 are x=0, y=3, and the pixels capturing the image of location A at time t=2 are x=4, y=3.

[0036] In the example shown in Figures 4(a) to (c), the pixel at x=0, y=3 at t=1 and the pixel at x=4, y=3 at t=2 are "pixels that image the same location on the workpiece W."

[0037] Therefore, if the polarity of both events is "+1", the signed polarity value "+1" is multiplied to obtain the numerical value "+2", and if the polarity of both events is "-1", the signed polarity value "-1" is multiplied to obtain the numerical value "-2".

[0038] Here, an example is shown in which the signed polarity values ​​of the received event data are accumulated for each pixel that images the same location on the workpiece W, but instead of accumulating the signed polarity values ​​of the events, the absolute values ​​of the polarity values ​​can also be accumulated. The absolute value of the polarity value means the number of times the event has occurred for each pixel that images the same location on the workpiece W.

[0039] When accumulating as absolute values, whether the polarity of both events is "+1" or "-1", the accumulated value will be "2". When using absolute values, you can convert each polarity value to an absolute value and then accumulate them, or you can accumulate the signed polarity values ​​and then convert them to absolute values ​​after accumulating.

[0040] 4(a) to 4(c) are explained using the example of two pieces of event data, but the amount of event data actually obtained within a predetermined imaging time is enormous. The information processing device 40 executes a process of accumulating the signed polarity values ​​or absolute values ​​of the polarity values ​​of all event data obtained within the predetermined imaging time for each pixel that images the same location on the workpiece W.

[0041] As mentioned above, event data is output as information such as (x, y, timestamp, polarity), and this alone does not allow one to identify which part of the work W the event occurred in; therefore, processing is required to link each event data to the part of the work W.

[0042] There are various methods for linking event data with parts of the workpiece W. For example, assuming that the movement of the workpiece W is a constant-velocity linear movement and the position of the EVS camera 30 is fixed, each piece of event data can be linked with a part of the workpiece W based on the movement speed v of the workpiece W and the movement time Δt of the workpiece W.

[0043] Specifically, the movement distance Δd of the work W during that movement time is calculated from the movement speed v of the work W and the movement time Δt of the work W, and by dividing the movement distance Δd by the unit pixel size, the number of pixels corresponding to the movement distance Δd can be calculated, and each event data can be linked to the part of the work W from this number of pixels.

[0044] The method of linking each event data with a part of the work W shown here is merely an example, and other linking methods are not excluded.

[0045] When the above-mentioned integration process is completed, the information processing device 40 executes a process of generating a two-dimensional histogram based on the values ​​obtained by the integration (histogram generation process) and a process of generating a static visualized image based on the generated two-dimensional histogram (image generation process). The static visualized image generated here is a color image or a monochrome image in which defective areas can be recognized.

[0046] After generating the static visualization image, the information processing device 40 outputs the generated static visualization image as data that can be displayed on a monitor mounted on (or connected to) the information processing device 40. By referring to the static visualization image displayed on the monitor, the worker can confirm which part of the workpiece W has a defect.

[0047] As examples of the generated static visualization images, FIG. 5(a) shows an example of a static visualization image of a workpiece W with no defects, and FIG. 5(b) shows an example of a static visualization image of a workpiece W with defects.

[0048] Looking at the area surrounded by a square in Figure 5(b), we can see changes that are not visible in the static visualization image in Figure 5(a). From this image, we can confirm that there is a defect in the area of ​​the actual workpiece W where the change appears.

[0049] Figure 6(a) is a graph of the integrated polarity values ​​of events that occurred on line L1 in Figure 5(a), and Figure 6(b) is a graph of the integrated polarity values ​​of events that occurred on line L2 in Figure 5(b). Looking at the area enclosed in a square in Figure 6(b), we can see a peak that does not appear in the static visualization image in Figure 6(a).

[0050] In the information processing system of the present application, prior to defect detection, it is necessary to perform initialization of output data, calibration of lenses, perspective correction of event data, etc. Lens calibration and perspective correction can be performed by existing or new methods.

[0051] (Example of use) Next, an example of how the information processing system of this embodiment is used will be described. (1) First, the worker places the workpiece W to be inspected on the transport body 10 and presses the start button displayed on the monitor of the information processing device 40. When the start button is pressed, the transport body 10 starts transporting the workpiece W, and the EVS camera 30 starts capturing images of the workpiece W during transport. (2) When the workpiece W passes through the predetermined imaging range of the EVS camera 30, the worker presses the stop button displayed on the monitor of the information processing device 40. When the stop button is pressed, imaging by the EVS camera 30 ends, and event data is acquired for the period from when the start button is pressed to when the stop button is pressed (predetermined imaging time). (3) After the image capturing is completed, the worker presses the data acquisition button displayed on the monitor of the information processing device 40. When the data acquisition button is pressed, the event data obtained by capturing the image with the EVS camera 30 is imported into the information processing device 40. (4) Once all event data obtained within the specified imaging time has been captured, the operator presses the 2D histogram generation button displayed on the monitor of the information processing device 40. When the 2D histogram generation button is pressed, a 2D histogram is generated in the information processing device 40 based on the event data. (5) After the two-dimensional histogram is generated, the worker presses a static visualized image generation button displayed on the monitor of the information processing device 40. When the static visualized image generation button is pressed, the information processing device 40 generates a static visualized image based on the two-dimensional histogram.

[0052] The static visualization image generated through the above steps is displayed on the monitor of the information processing device 40. The worker can identify the location of the defect in the workpiece W by visually checking the static visualization image displayed on the monitor.

[0053] In the above-mentioned usage example, the case where the workpieces W are inspected individually is taken as an example, but the information processing device 40 and the information processing system of the present application can also be introduced into part of a production line and configured to perform continuous defect detection. Also, the above-mentioned usage example is merely an example, and the display of the operation screen and the operation method can be designed appropriately according to the user.

[0054] In addition, in the above example of use, the operator manually starts and stops imaging, but it is also possible to provide sensors at the imaging start position and imaging stop position, and have imaging start and stop based on the detection signals of the sensors.

[0055] (Variation) The above-described embodiments and usage examples are merely examples, and the information processing device 40, information processing system, program, and information processing method of the present invention are not limited to these embodiments, etc. The inventions of the present application can be appropriately modified, such as by adding, replacing, or omitting components, within the scope of achieving the intended purpose. As an example, the following modifications are possible.

[0056] In the above embodiment, an example is given in which the workpiece W is moved by the conveying body 10, but the workpiece W and the EVS camera 30 only need to be in a relative moving relationship, and instead of the conveying body 10 that moves the workpiece W, a camera conveying body (not shown) that moves the EVS camera 30 can also be provided.

[0057] In some cases, a conveying body 10 for conveying the workpiece W and a camera conveying body (not shown) for moving the EVS camera 30 can be provided, so that the workpiece W is moved in the forward direction and the EVS camera 30 is moved in the reverse direction.

[0058] Although not specifically mentioned in the above embodiment, the information processing device 40 may be configured to add a predetermined value to the polarity integrated value when the integrated value exceeds a preset threshold value. This makes it possible to highlight defective areas on the generated two-dimensional histogram or static visualization image.

[0059] Although not specifically mentioned in the above embodiment, the information processing device 40 can also compare the generated static visualization image with a static visualization image generated in advance based on a defect-free workpiece W, and generate an image (difference image) showing the difference between the two. This makes it possible to identify defective locations even when comparing with a defect-free workpiece W.

[0060] In the above embodiment, an example is given in which the signed polarity values ​​or absolute values ​​of the polarity values ​​for both when the polarity is bright and when it is dark are accumulated, but the accumulation can also use the signed polarity values ​​or absolute values ​​of the polarity values ​​for either polarity.

[0061] In the above embodiment, as shown in Fig. 7(a), a case where reflected light is used is taken as an example, but a configuration using transmitted light is also possible. A configuration using transmitted light is particularly effective when the workpiece W is transparent (when the workpiece W transmits all or part of the light irradiated from the lighting fixture 20).

[0062] When using transmitted light, for example, as shown in Figure 7(b), the lighting device 20 and the EVS camera 30 can be placed at positions opposite each other across the workpiece W, so that the light emitted from the lighting device 20 and transmitted through the workpiece W can be received by the EVS camera 30.

[0063] In the above embodiment, an example is given in which the program is stored in storage within the information processing device 40, but the program may also be stored in a computer-readable recording medium, such as an external recording medium or an add-on built-in recording medium.

[0064] Although the above embodiment illustrates an example in which the information processing system is built in an on-premise environment, the information processing system can also be built on a cloud server in a cloud computing environment. In this case, the program is stored in a virtual server and executed on the cloud server.

[0065] The embodiments, use examples, and modifications disclosed in this application are merely examples and are not intended to limit the scope of the present invention. The technical scope of the present invention is defined by the claims. The technical scope of the present invention also includes equivalents to the claims. [Industrial Applicability]

[0066] The information processing device 40, the information processing system, the program, and the information processing method of the present invention can be suitably used for detecting defects in the workpiece W. [Explanation of symbols]

[0067] 10. Carrier 20 Lighting equipment 21 Translucent part 21a Masking material 30 EVS cameras 40 Information processing equipment double work

Claims

1. A process of accumulating the signed polarity value or the absolute value of the polarity value of the event data obtained by imaging the workpiece with an EVS camera for each pixel that images the same location of the workpiece; a process of generating a two-dimensional histogram based on the values ​​obtained by the integration; An information processing device that executes the above.

2. performing a process of generating a static visualization image based on the generated two-dimensional histogram; 2. The information processing device according to claim 1.

3. The static visualization image is a color image or a monochrome image.

3. The information processing device according to claim 2.

4. An EVS camera that captures images of the workpiece; an information processing device that processes event data obtained by the EVS camera; The information processing device is an information processing device according to any one of claims 1 to 3. Information processing system.

5. On the computer, A process of accumulating the signed polarity value or the absolute value of the polarity value of the event data obtained by imaging the workpiece with an EVS camera for each pixel that images the same location of the workpiece; a process of generating a two-dimensional histogram based on the values ​​obtained by the integration; A program that executes the following.

6. The information processing device A process of accumulating the signed polarity value or the absolute value of the polarity value of the event data obtained by imaging the workpiece with an EVS camera for each pixel that images the same location of the workpiece; a process of generating a two-dimensional histogram based on the values ​​obtained by the integration; An information processing method that performs the above.

Images