Object detection method and device and computer readable storage medium

Abstract

The embodiment of the invention provides an object detection method and device and a computer readable storage medium. The method comprises the following steps: respectively acquiring two-dimensional image data and three-dimensional image data of an object to be detected, wherein the two-dimensional image data and the three-dimensional image data have the same visual field; according to depth information of the three-dimensional image data, obtaining a region of interest in the visual field, and taking the region of interest in the visual field as a first template; obtaining a two-dimensional defect in the region of interest according to the two-dimensional image data and the first template, and taking a region without the two-dimensional defect in the region of interest as a second template; and performing three-dimensional defect detection on the three-dimensional image data according to the second template to obtain a three-dimensional defect in the region of interest.

Description

Technical Field

[0001] The present invention relates to the field of image analysis, and in particular to an object detection method, device and computer-readable storage medium, which are mainly used for defect detection on the surface of an object. Background Art

[0002] Surface defect detection of products is an important quality control process in the production process. In the detection of sheet metal or white bodies in stamping plants and body plants of automobile factories, as well as in the production of large metal parts that require real-time detection of target surfaces with different defects, the early methods of surface defect detection for two-dimensional defects (dirt, scratches, etc.) and three-dimensional defects (bumps, potholes, wrinkles, etc.) include: visual inspection with strong light, touch inspection with gloves, grinding with oilstone, etc. However, these methods are time-consuming and subjective, and largely depend on the concentration and attention of the inspectors. In fact, it is difficult to meet the needs of automated production lines.

[0003] In order to improve the degree of automation of product production lines and improve detection efficiency, surface defect detection based on optical two-dimensional / three-dimensional sensors has become a new trend. However, in the defect detection process, detection errors are often caused due to the inability to accurately distinguish and separately detect two-dimensional defects and three-dimensional defects.

[0004] Therefore, a method and device for detecting object defects more quickly and accurately is needed to avoid the problem of low detection accuracy caused by the inability to distinguish between two-dimensional defects and three-dimensional defects during defect detection. Summary of the invention

[0005] To solve the above technical problems, according to one aspect of the present invention, there is provided an object detection method, comprising: respectively acquiring two-dimensional image data and three-dimensional image data of an object to be detected, the two-dimensional image data and the three-dimensional image data having the same field of view; acquiring a region of interest in the field of view according to depth information of the three-dimensional image data, and using the region of interest in the field of view as a first template; acquiring two-dimensional defects in the region of interest according to the two-dimensional image data and the first template, and using a region in the region of interest that does not have two-dimensional defects as a second template; and performing three-dimensional defect detection on the three-dimensional image data according to the second template to acquire three-dimensional defects in the region of interest.

[0006] According to another aspect of the present invention, there is provided an object detection device, comprising: an acquisition unit, configured to respectively acquire two-dimensional image data and three-dimensional image data of an object to be detected, wherein the two-dimensional image data and the three-dimensional image data have the same field of view; a first template construction unit, configured to acquire a region of interest in the field of view according to depth information of the three-dimensional image data, and use the region of interest in the field of view as a first template; a second template construction unit, configured to acquire two-dimensional defects in the region of interest according to the two-dimensional image data and the first template, and use a region in the region of interest that does not have two-dimensional defects as a second template; and a detection unit, configured to perform three-dimensional defect detection on the three-dimensional image data according to the second template, and acquire three-dimensional defects in the region of interest.

[0007] According to another aspect of the present invention, there is provided an object detection device, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: respectively acquiring two-dimensional image data and three-dimensional image data of an object to be detected, the two-dimensional image data and the three-dimensional image data having the same field of view; acquiring a region of interest in the field of view based on depth information of the three-dimensional image data, and using the region of interest in the field of view as a first template; acquiring two-dimensional defects in the region of interest based on the two-dimensional image data and the first template, and using a region in the region of interest that does not have two-dimensional defects as a second template; and performing three-dimensional defect detection on the three-dimensional image data based on the second template to acquire three-dimensional defects in the region of interest.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: respectively obtain two-dimensional image data and three-dimensional image data of an object to be detected, the two-dimensional image data and the three-dimensional image data having the same field of view; obtain a region of interest in the field of view based on depth information of the three-dimensional image data, and use the region of interest in the field of view as a first template; obtain two-dimensional defects in the region of interest based on the two-dimensional image data and the first template, and use a region in the region of interest that does not have two-dimensional defects as a second template; perform three-dimensional defect detection on the three-dimensional image data based on the second template to obtain three-dimensional defects in the region of interest.

[0009] According to the above-mentioned object detection method, device and computer-readable storage medium of the present invention, it is possible to eliminate the interference of two-dimensional defects when detecting three-dimensional defects in the area of ​​interest by respectively collecting two-dimensional image data and three-dimensional image data of the object to be detected within the same field of view, thereby more clearly detecting and distinguishing two-dimensional defects and three-dimensional defects, improving the accuracy of defect detection, and improving the precision of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.

[0011] Figure 1 A flow chart showing an object detection method according to an embodiment of the present invention;

[0012] Figure 2 A schematic diagram of image acquisition of a camera and a projection device according to an example of an embodiment of the present invention is shown;

[0013] FIG3(a) shows a schematic diagram of white light projected by a projection device for acquiring two-dimensional image data according to an example of an embodiment of the present invention; FIG3(b) shows a schematic diagram of structured light projected by a projection device for acquiring three-dimensional image data according to an example of an embodiment of the present invention;

[0014] Figure 4 A schematic diagram showing two-dimensional image data within a field of view acquired for the component according to an example of an embodiment of the present invention;

[0015] FIG5(a) shows a schematic top view of three-dimensional image data within the field of view of the component obtained according to an example of an embodiment of the present invention, and FIG5(b) shows a schematic side view of three-dimensional image data within the field of view of the component obtained according to an example of an embodiment of the present invention;

[0016] Figure 6 A schematic diagram showing a first template according to an example of an embodiment of the present invention;

[0017] Figure 7 A schematic diagram showing a region to be detected of a two-dimensional defect according to an example of an embodiment of the present invention;

[0018] Figure 8 A schematic diagram showing a detection result of a two-dimensional defect according to an example of an embodiment of the present invention;

[0019] Fig. 9 A schematic diagram showing a method of removing two-dimensional defects from two-dimensional image data according to an example of an embodiment of the present invention;

[0020] Fig.10 A schematic diagram showing a second template according to an example of an embodiment of the present invention;

[0021] Fig.11 An example of performing defect detection on three-dimensional image data using a second template according to an example of an embodiment of the present invention is shown;

[0022] Fig.12 A schematic diagram showing the specific position and shape of a three-dimensional defect according to an example of an embodiment of the present invention;

[0023] Fig.13 A block diagram showing an object detection device according to an embodiment of the present invention;

[0024] Fig.14 A block diagram of an object detection device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] The object detection method, device and computer-readable storage medium according to the embodiments of the present invention will be described below with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements from beginning to end. It should be understood that the embodiments described herein are merely illustrative and should not be interpreted as limiting the scope of the present invention.

[0026] In the process of surface defect detection for products, defect detection based on optical sensors is widely used due to its non-contact and full coverage characteristics. However, in the process of surface detection based on optical two-dimensional / three-dimensional sensors, detection errors are often caused by the inability to distinguish between two-dimensional defects (dirt, scratches, etc.) and three-dimensional defects (bumps, pits, wrinkles, etc.).

[0027] Based on the above problems, embodiments of the present invention provide an object detection method, device and computer-readable storage medium.

[0028] Figure 1 A flow chart of an object detection method 100 according to an embodiment of the present invention is shown.

[0029] like Figure 1 As shown, in step S101, two-dimensional image data and three-dimensional image data of the object to be detected are respectively acquired, and the two-dimensional image data and the three-dimensional image data have the same field of view.

[0030] In an embodiment of the present invention, for an object to be detected, two-dimensional image data and three-dimensional image data in the same field of view can be captured by an image sensor respectively. Optionally, one image sensor can be used to collect two-dimensional image data and three-dimensional image data respectively; two-dimensional image data can also be collected by a two-dimensional image sensor respectively, and three-dimensional image data can be collected by a three-dimensional image sensor. In the acquisition process, the two-dimensional image data can be acquired by the image sensor by first projecting white light on the object to be detected by a projection device, and then the three-dimensional image data can be acquired by the image sensor by projecting structured light on the object to be detected by a projection device. In addition, optionally, the three-dimensional image data can be acquired by first projecting structured light on the object to be detected by a projection device, and then the two-dimensional image data can be acquired by the image sensor by projecting white light on the object to be detected by a projection device. The projection devices used to project white light and structured light can be the same or different, and the specific image data acquisition order is not limited. In addition, in another example, any other method can be adopted to acquire two-dimensional image data and three-dimensional image data according to the actual application scenario. For example, two-dimensional image data can be acquired by projecting light of a color other than white, or three-dimensional image data can be acquired by using a binocular camera, a depth-of-field camera, etc., all of which are not limited here. During the data acquisition process, calibration can be performed to make the acquired two-dimensional image data and three-dimensional image data have the same field of view.

[0031] In step S102, a region of interest in the field of view is acquired according to the depth information of the three-dimensional image data, and the region of interest in the field of view is used as a first template.

[0032] In this step, the three-dimensional image data can be optionally divided according to the depth information of the three-dimensional image to obtain the region of interest in the field of view. Specifically, considering that the depths of the foreground area and the background area in the three-dimensional image data are different, the foreground area of ​​the three-dimensional image data can be extracted based on the depth segmentation algorithm as the region of interest in the field of view, and as the first template subsequently used for the two-dimensional image data. In particular, when the colors of the foreground area and the background area in the field of view are similar, extracting the region of interest based on the depth information of the three-dimensional image data can accelerate the subsequent processing and detection of two-dimensional defects in the two-dimensional image data.

[0033] In step S103, two-dimensional defects in the region of interest are acquired according to the two-dimensional image data and the first template, and a region in the region of interest without two-dimensional defects is used as a second template.

[0034] In this step, optionally, the first template obtained previously can be used to define a corresponding region of interest for the two-dimensional image data, and two-dimensional defect detection can be performed on the region of interest defined by the first template. Optionally, the region of interest of the two-dimensional image data can be detected in a variety of ways, such as one or more ways based on threshold segmentation, edge detection, filtering algorithm, deep learning, etc., so as to obtain two-dimensional defects in the region of interest.

[0035] For example, when the area of ​​interest obtained by using the depth data of the three-dimensional image data is the foreground area, various defect detection methods can be used in this step to perform two-dimensional defect detection on the foreground area in the two-dimensional image data to obtain two-dimensional defects in the foreground area in the field of view.

[0036] After detecting the two-dimensional defects in the region of interest, a second template can be generated using the first template representing the region of interest and the detected two-dimensional defects. Specifically, the second template can be obtained by performing an "AND operation" on the first template and the non-two-dimensional defect region, that is, the region in the region of interest that does not have two-dimensional defects is used as the second template.

[0037] In step S104, three-dimensional defect detection is performed on the three-dimensional image data according to the second template to obtain three-dimensional defects in the region of interest.

[0038] In this step, three-dimensional defect detection can be performed in the area corresponding to the second template in the three-dimensional image data based on the second template. Specifically, three-dimensional defect detection can be performed based on the depth information of the three-dimensional image data to obtain three-dimensional defects.

[0039] By performing 3D defect detection in a region of interest without 2D defects in 3D image data as described above, invalid 3D image data can be removed to speed up the defect detection process of 3D defects.

[0040] In addition, in the present method, after obtaining the two-dimensional defects and three-dimensional defects in the region of interest in the field of view, the detected two-dimensional defects and three-dimensional defects can be extracted and displayed respectively. For example, the specific information of the obtained two-dimensional defects and three-dimensional defects can include the information of the position and / or shape of the two-dimensional defects and three-dimensional defects, and after obtaining the above information, the information of the position and / or shape of the obtained two-dimensional defects and three-dimensional defects can be extracted, displayed, etc., or the relevant data of the information such as the position and shape of the two-dimensional defects and three-dimensional defects can be output to other devices, such as printers, projectors, etc. Further, different classification results of different defects can be shown accordingly, for example, defects can be classified into two-dimensional defects and three-dimensional defects, etc. Of course, various information in the image can be further used to finely classify defects into different types, for example, the depth information in the three-dimensional image can be used to distinguish between convexities and concavities in three-dimensional defects, etc.

[0041] According to the above-mentioned object detection method of the embodiment of the present invention, by respectively collecting two-dimensional image data and three-dimensional image data of the object to be detected within the same field of view, the interference of two-dimensional defects can be eliminated when detecting three-dimensional defects in the area of ​​interest, so that two-dimensional defects and three-dimensional defects can be detected and distinguished more clearly, the accuracy of defect detection can be improved, and the precision of defect detection can be improved.

[0042] A specific example of the object detection method according to an embodiment of the present invention is described below. In this example, the object to be detected may be a specific component on a vehicle, and in the object detection method in this example, two-dimensional and three-dimensional defect detection may be performed on the component on the vehicle to judge and control the quality of the component.

[0043] Specifically, in this example, two-dimensional image data and three-dimensional image data of the object to be detected may be first acquired respectively, wherein the two-dimensional image data and the three-dimensional image data have the same field of view.

[0044] In this example, a camera can be used as an image sensor to collect the two-dimensional image data obtained by projecting white light onto the component through a projection device, and then collect the three-dimensional image data obtained by projecting structured light onto the component through the projection device. During the image data acquisition process, the camera and the projection device can be calibrated so that the two cover the same area, so that the collected two-dimensional image data and three-dimensional image data have the same field of view.

[0045] Figure 2A schematic diagram of image acquisition of a camera and a projection device according to an example of an embodiment of the present invention is shown. FIG3(a) shows a schematic diagram of white light projected by a projection device for acquiring two-dimensional image data according to an example of an embodiment of the present invention; FIG3(b) shows a schematic diagram of structured light projected by a projection device for acquiring three-dimensional image data according to an example of an embodiment of the present invention. Figure 2 When the projection device shown in FIG. 3( a ) projects uniform white light as shown in FIG. 3( a ) onto the component, the camera photographs the component to obtain corresponding two-dimensional image data. Figure 4 A schematic diagram showing two-dimensional image data within a field of view acquired for the component according to an example of an embodiment of the present invention is shown. Figure 4 The two-dimensional image data in shows the two-dimensional grayscale information on the component, which shows the distribution and changes of grayscale values ​​at different positions within the field of view of the component.

[0046] In addition, when the projection device projects the sinusoidal structured light as shown in FIG. 3( b) to the component, the camera shoots the component to obtain corresponding three-dimensional image data. Specifically, the three-dimensional image data in the field of view can be obtained, for example, by using a typical single-frame three-dimensional reconstruction method-Fourier transform profilometry (FTP). The processing process of the three-dimensional reconstruction method based on Fourier transform profilometry may include Fourier transform, filtering, inverse Fourier transform, phase calculation, phase unwrapping and height calculation for the image obtained by the camera when projecting the sinusoidal structured light. After these processes, the three-dimensional image data of the component can be obtained, which includes the three-dimensional information in the component. FIG. 5( a) shows a schematic top view of the three-dimensional image data in the field of view of the component obtained according to an example of an embodiment of the present invention, and FIG. 5( b) shows a schematic side view of the three-dimensional image data in the field of view of the component obtained according to an example of an embodiment of the present invention. The three-dimensional image data in FIG. 5( a) and FIG. 5( b) show the three-dimensional depth information on the component, which shows the distribution and change of the three-dimensional depth values ​​at different positions within the field of view of the component.

[0047] In this example, the method and sequence for acquiring two-dimensional image data and three-dimensional image data are examples and are not limited in actual applications.

[0048] Subsequently, the region of interest in the field of view may be acquired according to the depth information of the three-dimensional image data, and the region of interest in the field of view may be used as the first template.

[0049] Taking into account the different depths of the foreground area and the background area in the three-dimensional image data, the depth information in the three-dimensional image data can be used to extract the foreground area of ​​the three-dimensional image data based on the depth segmentation algorithm as the area of ​​interest in the field of view, and as the first template subsequently used for the two-dimensional image data. Figure 6FIG. 5 is a schematic diagram of a first template according to an example of an embodiment of the present invention. When the depths of the foreground area and the background area of ​​the component shown in the three-dimensional image data of FIG. 5( a) and FIG. 5( b) are different, the three-dimensional image data can be depth segmented to obtain the following: Figure 6 A first template is shown that includes only the foreground area shown as the white portion of the component as the region of interest.

[0050] In the field of view, the colors of the foreground and background areas are similar (such as Figure 4 In the case of a grayscale value shown in FIG. 1 , extracting a region of interest based on depth information of three-dimensional image data can accelerate the subsequent processing and detection of two-dimensional defects in two-dimensional image data.

[0051] After acquiring the first template, the two-dimensional defect in the region of interest may be acquired according to the two-dimensional image data and the first template. Figure 7 Shows the Figure 4 The two-dimensional image data shown is based on Figure 6 The first template shown is a schematic diagram of dividing the area to be detected of two-dimensional defects. Figure 7 In the Figure 4 The two-dimensional defect detection is performed on the area defined by the first template in the two-dimensional image data to obtain the detection result of the two-dimensional defect. Optionally, the two-dimensional defect detection can be performed on the area of ​​interest of the two-dimensional image data in a variety of ways, for example, the two-dimensional defects in the area of ​​interest can be obtained based on various two-dimensional defect detection methods such as threshold segmentation, edge detection, filtering algorithm, deep learning, etc. Figure 8 Shows the Figure 4 The schematic diagram of the result of two-dimensional defect detection using two-dimensional image data shown in FIG. Figure 8 The two irregular regions show the positions of the two-dimensional defects in the region of interest. The two-dimensional defect detection performed on the region of interest based on the first template can speed up the detection and acquisition process of the two-dimensional defects.

[0052] After detecting the two-dimensional defects in the region of interest, the second template can be generated using the first template and the detected two-dimensional defects. Specifically, the second template can be obtained by performing an "AND operation" on the first template and the non-two-dimensional defect region, that is, the region without two-dimensional defects in the region of interest is used as the second template. Fig. 9 A schematic diagram showing removal of two-dimensional defects in two-dimensional image data according to an example of an embodiment of the present invention. Fig. 9 The white part in the image is the area without two-dimensional defects in the two-dimensional image data. Fig. 9 After removing the defect area, the field of view can be compared with Figure 6 The first template shown is ANDed to obtain the second template. Fig.10A schematic diagram showing a second template according to an example of an embodiment of the present invention is shown. Fig.10 In the figure, the portion of the region of interest shown in white from which two-dimensional defects are removed is the region defined by the second template for subsequent three-dimensional defect detection.

[0053] After acquiring the second template for the component, three-dimensional defect detection may be performed on the three-dimensional image data according to the second template to acquire three-dimensional defects in the region of interest.

[0054] Specifically, it can be based on Fig.10 The second template shown in FIG. 5( a ) performs three-dimensional defect detection on the three-dimensional image data within the range of the second template to obtain a three-dimensional defect detection result. Fig.11 An example of using a second template to detect defects in three-dimensional image data according to an example of an embodiment of the present invention is shown. A three-dimensional defect schematic diagram represented by dark gray can be obtained within the field of view defined by the second template in the three-dimensional image data. Optionally, three-dimensional defect detection can be performed based on the depth information of the three-dimensional image data to obtain three-dimensional defects. Fig.12 The specific position and shape of the three-dimensional defect detected in the three-dimensional image data of the example according to the embodiment of the present invention are shown. The three-dimensional defect detection process based on the area defined by the second template can remove invalid three-dimensional information in the acquired three-dimensional image data to further accelerate the three-dimensional defect detection process.

[0055] After respectively acquiring the two-dimensional defects and three-dimensional defects in the region of interest within the field of view, the detected two-dimensional defects and three-dimensional defects can be extracted and displayed respectively. For example, the specific information of the acquired two-dimensional defects and three-dimensional defects may include the information of the position and / or shape of the two-dimensional defects and three-dimensional defects, and after acquiring the above information, the acquired information of the position and / or shape of the two-dimensional defects and three-dimensional defects can be extracted, displayed, etc., or the relevant data of the information of the position, shape, etc. of the two-dimensional defects and three-dimensional defects can be output to other devices, such as printers, projectors, etc. In this example, for example, Fig.12The specific position and shape of the three-dimensional defect shown in the image are displayed on the display, or the relevant information can be further output to a printer or projector. In addition, different classification results of different defects can be shown accordingly, for example, defects can be classified into two-dimensional defects and three-dimensional defects, etc., and various information in the image can be further used on this basis to finely classify defects into different types. For example, the depth information in the three-dimensional image can be used to distinguish between convexities and concavities in three-dimensional defects. Optionally, a defect whose depth information of a three-dimensional defect is greater than the average depth of the region of interest by more than a certain threshold can be a convexity, and a defect whose depth information is less than the average depth of the region of interest by more than a certain threshold can be a concavity.

[0056] The method according to the example of the embodiment of the present invention can perform 3D image reconstruction and 2D image acquisition based on a single-frame structured image. In the defect detection process, only two images need to be captured for a certain scene or component (the capture time can be <10ms) to quickly obtain 2D / 3D image data, which is very practical for defect detection of real-time moving objects.

[0057] In specific two-dimensional defect and three-dimensional defect processes, since the exemplary method of the embodiment of the present invention can cleverly use the captured two-dimensional image data and three-dimensional image data to delineate the corresponding detection area, the processing time of the defect detection process is greatly accelerated, and the mutual influence between defects of different categories and the resulting detection errors are effectively avoided.

[0058] Below, refer to Fig.13 The object detection device according to the embodiment of the present invention is described. Fig.13 FIG. 1 is a block diagram of an object detection device 1300 according to an embodiment of the present invention. Fig.13 As shown in FIG. 1 , the object detection device 1300 includes an acquisition unit 1310, a first template construction unit 1320, a second template construction unit 1330, and a detection unit 1340. In addition to these units, the object detection device 1300 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the object detection device 1300 according to the embodiment of the present invention are the same as those described above with reference to FIG. Figure 1 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0059] Fig.13 The acquisition unit 1310 of the object detection device 1300 in the embodiment respectively acquires the two-dimensional image data and the three-dimensional image data of the object to be detected, and the two-dimensional image data and the three-dimensional image data have the same field of view.

[0060] In an embodiment of the present invention, for an object to be detected, the acquisition unit 1310 may capture two-dimensional image data and three-dimensional image data in the same field of view through an image sensor, respectively. Optionally, one image sensor may be used to collect two-dimensional image data and three-dimensional image data, respectively; two-dimensional image data may also be collected by a two-dimensional image sensor, respectively, and three-dimensional image data may be collected by a three-dimensional image sensor. During the acquisition process, the two-dimensional image data may be first acquired through an image sensor by projecting white light on the object to be detected through a projection device, and then the three-dimensional image data may be acquired through an image sensor by projecting structured light on the object to be detected through a projection device. In addition, optionally, the three-dimensional image data may be first acquired through an image sensor by projecting structured light on the object to be detected through a projection device, and then the two-dimensional image data may be acquired through an image sensor by projecting white light on the object to be detected through a projection device. The projection devices used to project white light and structured light may be the same or different, and the specific image data acquisition order is not limited. In addition, in another example, any other method may be adopted to acquire two-dimensional image data and three-dimensional image data according to the actual application scenario. For example, two-dimensional image data can be acquired by projecting light of a color other than white, or three-dimensional image data can be acquired by using a binocular camera, a depth-of-field camera, etc., all of which are not limited here. During the data acquisition process, calibration can be performed to make the acquired two-dimensional image data and three-dimensional image data have the same field of view.

[0061] The first template constructing unit 1320 obtains a region of interest in the field of view according to the depth information of the three-dimensional image data, and uses the region of interest in the field of view as a first template.

[0062] Optionally, the first template construction unit 1320 can divide the three-dimensional image data according to the depth information of the three-dimensional image to obtain the region of interest in the field of view. Specifically, considering the different depths of the foreground area and the background area in the three-dimensional image data, the foreground area of ​​the three-dimensional image data can be extracted based on the depth segmentation algorithm as the region of interest in the field of view, and as the first template subsequently used for the two-dimensional image data. In particular, when the colors of the foreground area and the background area in the field of view are similar, extracting the region of interest based on the depth information of the three-dimensional image data can accelerate the subsequent processing and detection of two-dimensional defects in the two-dimensional image data.

[0063] The second template constructing unit 1330 obtains the two-dimensional defects in the region of interest according to the two-dimensional image data and the first template, and uses the region of the region of interest without the two-dimensional defects as the second template.

[0064] Optionally, the second template construction unit 1330 may define a corresponding region of interest for the two-dimensional image data using the first template previously acquired, and perform two-dimensional defect detection on the region of interest of the two-dimensional image data defined by the first template. Optionally, the region of interest of the two-dimensional image data may be detected in a variety of ways, such as one or more ways based on threshold segmentation, edge detection, filtering algorithm, deep learning, etc., so as to obtain the two-dimensional defects in the region of interest.

[0065] For example, when the area of ​​interest obtained by using the depth data of the three-dimensional image data is the foreground area, the second template construction unit 1330 can use various defect detection methods to perform two-dimensional defect detection on the foreground area in the two-dimensional image data to obtain two-dimensional defects in the foreground area in the field of view.

[0066] After detecting the two-dimensional defects in the region of interest, a second template can be generated using the first template representing the region of interest and the detected two-dimensional defects. Specifically, the second template can be obtained by performing an "AND operation" on the first template and the non-two-dimensional defect region, that is, the region in the region of interest that does not have two-dimensional defects is used as the second template.

[0067] The detection unit 1340 performs 3D defect detection on the 3D image data according to the second template to obtain the 3D defects in the region of interest.

[0068] The detection unit 1340 may perform 3D defect detection in the area corresponding to the second template in the 3D image data based on the second template. Specifically, the 3D defect detection may be performed according to the depth information of the 3D image data to obtain the 3D defect.

[0069] By performing 3D defect detection in a region of interest without 2D defects in 3D image data as described above, invalid 3D image data can be removed to speed up the defect detection process of 3D defects.

[0070] In addition, in the present device, after respectively acquiring the two-dimensional defects and three-dimensional defects in the region of interest within the field of view, the detection unit 1340 can also extract and display the detected two-dimensional defects and three-dimensional defects respectively. For example, the specific information of the acquired two-dimensional defects and three-dimensional defects can include the information of the position and / or shape of the two-dimensional defects and the three-dimensional defects, and after acquiring the above information, the acquired information of the position and / or shape of the two-dimensional defects and the three-dimensional defects can be extracted, displayed, etc., or the relevant data of the information of the position, shape, etc. of the two-dimensional defects and the three-dimensional defects can be output to other devices, such as printers, projectors, etc. Further, different classification results of different defects can be shown accordingly, for example, defects can be classified into two-dimensional defects and three-dimensional defects, etc. Of course, various information in the image can be further used to finely classify defects into different types, for example, the depth information in the three-dimensional image can be used to distinguish between convexities and concavities in three-dimensional defects, etc.

[0071] According to the above-mentioned object detection device of the embodiment of the present invention, it is possible to eliminate the interference of two-dimensional defects when detecting three-dimensional defects in the area of ​​interest by respectively collecting two-dimensional image data and three-dimensional image data of the object to be detected within the same field of view, thereby being able to more clearly detect and distinguish two-dimensional defects and three-dimensional defects, thereby improving the accuracy of defect detection and the precision of defect detection.

[0072] Below, refer to Fig.14 The object detection device according to the embodiment of the present invention is described. Fig.14 FIG. 1 is a block diagram of an object detection device 1400 according to an embodiment of the present invention. Fig.14 As shown, the device 1400 may be a computer or a server.

[0073] like Fig.14 As shown, the object detection device 1400 includes one or more processors 1410 and a memory 1420. Of course, in addition to this, the object detection device 1400 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Fig.14 The components and structures of the object detection device 1400 shown are merely exemplary and non-limiting. The object detection device 1400 may also have other components and structures as required.

[0074] The processor 1410 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1420 to execute desired functions, so that the processor 1410 can control the robotic arm, and the steps executed by the processor 1410 may include: respectively acquiring two-dimensional image data and three-dimensional image data of the object to be detected, the two-dimensional image data and the three-dimensional image data having the same field of view; acquiring a region of interest in the field of view based on depth information of the three-dimensional image data, and using the region of interest in the field of view as a first template; acquiring two-dimensional defects in the region of interest based on the two-dimensional image data and the first template, and using a region in the region of interest that does not have two-dimensional defects as a second template; performing three-dimensional defect detection on the three-dimensional image data based on the second template to acquire three-dimensional defects in the region of interest.

[0075] The memory 1420 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1410 may execute the program instructions to implement the functions of the object detection device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the object detection method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.

[0076] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions implement the following steps when executed by a processor: respectively acquiring two-dimensional image data and three-dimensional image data of an object to be detected, the two-dimensional image data and the three-dimensional image data having the same field of view; acquiring a region of interest in the field of view based on depth information of the three-dimensional image data, and using the region of interest in the field of view as a first template; acquiring two-dimensional defects in the region of interest based on the two-dimensional image data and the first template, and using a region in the region of interest that does not have two-dimensional defects as a second template; and performing three-dimensional defect detection on the three-dimensional image data based on the second template to acquire three-dimensional defects in the region of interest.

[0077] Of course, the above-mentioned specific embodiments are merely examples rather than limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.

[0078] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0079] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0080] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to a singular element, such as using the article "a", "an", or "the", is not to be construed as limiting the element to the singular.

[0081] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0082] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0083] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0084] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.

[0085] The method of the invention herein includes one or more actions for implementing the method described. The method and / or actions can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.

[0086] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.

[0087] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

[0088] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.

[0089] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0090] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0091] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0092] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.

[0093] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An object detection method, comprising: Respectively acquiring two-dimensional image data and three-dimensional image data of the object to be detected, wherein the two-dimensional image data and the three-dimensional image data have the same field of view; Acquire a region of interest in the field of view according to the depth information of the three-dimensional image data, and use the region of interest in the field of view as a first template; Acquire the two-dimensional defects in the region of interest according to the two-dimensional image data and the first template, and use the region of the region of interest without the two-dimensional defects as a second template; Perform three-dimensional defect detection on the three-dimensional image data according to the second template to obtain three-dimensional defects in the region of interest.

2. The method of claim 1, wherein: Respectively acquiring two-dimensional image data and three-dimensional image data of the object to be detected includes: Acquiring the two-dimensional image data by projecting white light onto the object to be detected; and / or The three-dimensional image data is acquired by projecting structured light onto the object to be detected.

3. The method of claim 1, wherein: Acquiring the region of interest in the field of view according to the depth information of the three-dimensional image data includes: A foreground area of ​​the three-dimensional image data is extracted based on a depth segmentation algorithm as a region of interest in the field of view.

4. The method of claim 1, wherein: Acquiring the two-dimensional defect in the region of interest according to the two-dimensional image data and the first template includes: Based on one or more methods of threshold segmentation, edge detection, filtering algorithm, and deep learning, two-dimensional defect detection is performed on the two-dimensional image data in the region of interest defined by the first template to obtain the two-dimensional defects in the region of interest.

5. The method of claim 1, wherein: Performing three-dimensional defect detection on the three-dimensional image data according to the second template to obtain the three-dimensional defects in the region of interest includes: According to the depth information of the three-dimensional image data, three-dimensional defect detection is performed on the three-dimensional image data in a region of interest defined by the second template and having no two-dimensional defects, so as to obtain the three-dimensional defect.

6. The method of claim 1, wherein: The method further comprises: The two-dimensional defects and three-dimensional defects in the detected region of interest are extracted respectively, and the defect classification results are displayed.

7. The method according to any one of claims 1 to 6, wherein: Acquiring the two-dimensional defect and / or the three-dimensional defect in the region of interest includes: The position and / or shape of the two-dimensional defect and / or three-dimensional defect in the region of interest is obtained, and at least one of the following operations is performed on the obtained position and / or shape of the two-dimensional defect and / or three-dimensional defect, including extraction, display, and output to other devices.

8. An object detection device, comprising: an acquisition unit, configured to respectively acquire two-dimensional image data and three-dimensional image data of the object to be detected, wherein the two-dimensional image data and the three-dimensional image data have the same field of view; A first template construction unit is configured to obtain a region of interest in the field of view according to the depth information of the three-dimensional image data, and use the region of interest in the field of view as a first template; a second template construction unit configured to obtain the two-dimensional defects in the region of interest according to the two-dimensional image data and the first template, and use the region of the region of interest without the two-dimensional defects as a second template; The detection unit is configured to perform three-dimensional defect detection on the three-dimensional image data according to the second template to obtain the three-dimensional defects in the region of interest.

9. An object detection device, comprising: processor; and a memory having computer program instructions stored therein, Wherein, when the computer program instructions are executed by the processor, the processor is enabled to control the robotic arm, and the processor is capable of performing the following steps: Respectively acquiring two-dimensional image data and three-dimensional image data of the object to be detected, wherein the two-dimensional image data and the three-dimensional image data have the same field of view; Acquire a region of interest in the field of view according to the depth information of the three-dimensional image data, and use the region of interest in the field of view as a first template; Acquire the two-dimensional defects in the region of interest according to the two-dimensional image data and the first template, and use the region of the region of interest without the two-dimensional defects as a second template; Perform three-dimensional defect detection on the three-dimensional image data according to the second template to obtain three-dimensional defects in the region of interest.

10. A computer-readable storage medium having computer program instructions stored thereon, wherein: When the computer program instructions are executed by a processor, the following steps are implemented: Respectively acquiring two-dimensional image data and three-dimensional image data of the object to be detected, wherein the two-dimensional image data and the three-dimensional image data have the same field of view; Acquire a region of interest in the field of view according to the depth information of the three-dimensional image data, and use the region of interest in the field of view as a first template; Acquire the two-dimensional defects in the region of interest according to the two-dimensional image data and the first template, and use the region of the region of interest without the two-dimensional defects as a second template; Perform three-dimensional defect detection on the three-dimensional image data according to the second template to obtain three-dimensional defects in the region of interest.

Images