Defect detection method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a defect detection method and device, electronic equipment, a storage medium and a program product. The defect detection method comprises the steps that shot images and point cloud images of a detection object in multiple view fields are acquired; processing the point cloud image to obtain first defect information; second defect information is determined on the basis of a transformed shot image obtained by processing the shot image, and the first defect information and the second defect information aim at different defect types; and determining a defect detection result according to the first defect information and the second defect information on the basis of a mapping relationship between the position information of the transformed shot image and the point cloud image.

Description

Technical Field

[0001] The present disclosure relates to the fields of image processing, point cloud processing and industrial technology, and more specifically, to a defect detection method, device, electronic device, storage medium and program product. Background Art

[0002] Defect detection is the process of identifying and evaluating possible flaws, damage or defects on the object being tested during the production process. Defect detection is increasingly being used in a variety of fields, such as industrial manufacturing, material science and quality control.

[0003] Through defect detection, defects on the inspection object can be discovered and resolved early, ensuring product quality and reducing possible problems in subsequent applications. Therefore, how to more effectively improve the comprehensiveness and accuracy of defect detection is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the present disclosure provides a defect detection method, apparatus, electronic device, storage medium and program product.

[0005] According to one aspect of the present disclosure, a defect detection method is provided, including: acquiring captured images and point cloud images of a detection object in multiple fields of view; determining first defect information based on at least one candidate defect area obtained by processing the point cloud image; determining second defect information based on a transformed captured image obtained by processing the captured image, the first defect information and the second defect information being for different defect types; and determining a defect detection result based on the first defect information and the second defect information based on a mapping relationship between respective position information of the transformed captured image and the point cloud image.

[0006] According to another aspect of the present disclosure, a defect detection device is provided, including an acquisition module for acquiring captured images and point cloud images of a detection object in multiple fields of view; a first determination module for determining first defect information based on at least one candidate defect area obtained by processing the point cloud image; a second determination module for determining second defect information based on a transformed captured image obtained by processing the captured image, the first defect information and the second defect information being for different defect types; and a third determination module for determining a defect detection result based on the first defect information and the second defect information, based on the relationship between the transformed captured image and the point cloud image.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described in the present disclosure.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which executable instructions are stored. When the executable instructions are executed by a processor, the processor implements the method described in the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided. The computer program product includes computer executable instructions. When the computer executable instructions are executed, they are used to implement the method described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0011] Figure 1 The system architecture to which the defect detection method according to the embodiment of the present disclosure can be applied is schematically shown;

[0012] Figure 2 A flowchart of a defect detection method according to an embodiment of the present disclosure is schematically shown;

[0013] Figure 3 An example schematic diagram of a process of determining multiple candidate defect regions based on multiple candidate point cloud blocks obtained by dividing a point cloud image according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 4 An exemplary schematic diagram schematically shows a process of screening multiple merged defect regions based on screening rules to obtain first defect information according to an embodiment of the present disclosure;

[0015] Figure 5 An example schematic diagram of a maximum distance determination process according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 6 An example schematic diagram schematically shows a target detection model training process according to an embodiment of the present disclosure;

[0017] Figure 7 An example schematic diagram schematically shows a target detection model application process according to an embodiment of the present disclosure;

[0018] Figure 8 A block diagram schematically shows a defect detection device according to an embodiment of the present disclosure; and

[0019] Fig. 9 A block diagram of an electronic device suitable for implementing a defect detection method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0020] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0021] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0022] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0023] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0024] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0025] In one example, defect detection of the inspection object can be performed by visual inspection, ultrasonic inspection, and magnetic particle inspection. Visual inspection refers to the use of cameras and image processing algorithms to detect defects on the surface of the inspection object. Ultrasonic inspection refers to the use of ultrasonic technology to detect defects on the surface of the inspection object. Magnetic particle inspection refers to the use of magnetic powder to spread on the surface of the inspection object and detect defects where the magnetic powder is deposited.

[0026] However, since the overall detection process of the above defect detection method requires human participation, the degree of automation is low; and the defect detection results obtained based on the above method are usually for a certain type of defect, so the detection accuracy is relatively limited. Therefore, how to achieve automated defect detection and improve the comprehensiveness and accuracy of defect detection results is an urgent problem to be solved.

[0027] To this end, the present disclosure provides a defect detection method, device, electronic device, storage medium and program product, which can be applied to the fields of image processing, point cloud processing and industrial technology. The defect detection method includes: acquiring captured images and point cloud images of the detection object under multiple fields of view; determining first defect information based on at least one candidate defect area obtained by processing the point cloud image; determining second defect information based on a transformed captured image obtained by processing the captured image, the first defect information and the second defect information being for different defect types; and determining a defect detection result based on the first defect information and the second defect information based on a mapping relationship between the position information of the transformed captured image and the point cloud image.

[0028] Figure 1 The system architecture to which the defect detection method according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between different devices.

[0030] It should be noted that the defect detection method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the defect detection device provided in the embodiment of the present disclosure can generally be set in the server 105.

[0031] Alternatively, the defect detection method provided in the embodiment of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the defect detection apparatus provided in the embodiment of the present disclosure may also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0032] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0034] The above describes the system architecture to which the defect detection method provided by the present disclosure can be applied. Figure 2 For example, the defect detection process of the present disclosure is further described.

[0035] Figure 2 The flowchart of the defect detection method according to the embodiment of the present disclosure is schematically shown.

[0036] like Figure 2 As shown, the defect detection method 200 includes operations S210 to S240.

[0037] In operation S210 , captured images and point cloud images of the detection object in multiple fields of view are acquired.

[0038] In operation S220, the point cloud image is processed to obtain first defect information.

[0039] In operation S230, second defect information is determined based on a transformed captured image obtained by processing the captured image, the first defect information and the second defect information being for different defect types.

[0040] In operation S240 , a defect detection result is determined according to the first defect information and the second defect information based on a mapping relationship between respective position information of the transformed photographed image and the point cloud image.

[0041] The detection object may refer to a target or object that needs to be paid attention to during the defect detection process. The method of acquiring captured images and point cloud images can be configured according to actual business needs and is not limited here. For example, the method of acquiring captured images may include at least one of the following: an acquisition method based on a digital camera, an acquisition method based on scanning, and an acquisition method based on video capture. Alternatively, the method of acquiring point cloud data may include at least one of the following: an acquisition method based on laser radar, an acquisition method based on structured light projection, and an acquisition method based on stereo matching of binocular or multi-cameras, etc.

[0042] The point cloud image may refer to data composed of a series of second position information on the surface of the detection object. The point cloud image represents the external shape, structure and geometric information of the detection object. The point cloud image includes multiple points. The second position information may represent the position of each point relative to a reference origin in three-dimensional space.

[0043] In one example, after obtaining a point cloud image, defect detection can be performed on the point cloud image to obtain first defect information. The first defect information may include a defect location and a defect type. The specific method of performing defect detection on the point cloud image can be configured according to actual business needs and is not limited here. For example, a defect detection method based on a point cloud image may include a method based on feature extraction, a method based on clustering, a method based on machine learning, and a method based on local descriptors.

[0044] The captured image may refer to a two-dimensional image or photo captured by a video camera or a camera, and the captured image may include data consisting of a series of first position information on the surface of the detection object. The captured image includes multiple points. The first position information may represent the position of each point in the two-dimensional plane relative to a certain reference origin.

[0045] In one example, after obtaining the captured image, the captured image may be transformed to obtain a transformed captured image. The purpose of the transformation process is to change the appearance or content of the captured image. The specific method of the transformation process can be configured according to actual business needs and is not limited here. For example, the transformation process may include scaling, rotation, flipping, cropping, deformation, perspective transformation, and affine transformation.

[0046] After obtaining the transformed captured image, defect detection can be performed on the transformed captured image to obtain second defect information. The second defect information may include defect location and defect type. The specific method of performing defect detection on the transformed captured image can be configured according to actual business needs and is not limited here. For example, defect detection methods based on the transformed captured image may include methods based on feature extraction, methods based on machine learning, and methods based on target detection.

[0047] In one example, the shape of the detection object can be configured according to actual business needs and is not limited here. For example, the shape of the detection object can include at least one of the following: a sphere, a cube, a cylinder, a cone, a polyhedron, and an irregular shape. In one example, the spherical detection object can include a workpiece with a spherical or spherical shape, which can be a complete sphere or a partial sphere, such as a spherical surface and a hemispherical surface.

[0048] It should be noted that the first defect information and the second defect information are for different defect types. Defect type refers to the classification of defects included in the detection object. The defect types targeted by the first defect information and the second defect information can be configured according to actual business needs and are not limited here. For example, the first defect information can be for defects with depth information, and the second defect information can be for defects located on the surface of the detection object. For example, the defect type targeted by the first defect information may include at least one of the following: scratches, cracks, fissures, holes, pores, voids, sediments or foreign matter, etc. Alternatively, the defect type targeted by the second defect information may include at least one of the following: stains, scratches, rust, corrosion, burrs or coating peeling, etc.

[0049] After obtaining the first defect information and the second defect information, the mapping relationship between the transformed captured image and the point cloud image can be determined. The mapping relationship characterizes the correspondence between the first position information in the transformed captured image and the second position information in the point cloud image. Based on the mapping relationship, the defect position included in the first defect information and the defect position included in the second defect information can be uniformly converted into the same three-dimensional model to obtain a defect detection result. The defect detection result can include the defect position and defect type in the three-dimensional space. The defect detection result includes both defects with depth information and defects located on the surface of the detection object.

[0050] According to the embodiments of the present disclosure, by acquiring captured images and point cloud images of the inspection object under multiple fields of view, all-round data collection is achieved, which helps to achieve more comprehensive defect detection. By performing defect detection on the captured images and point cloud images respectively, defect information for different defect types can be obtained, and two-dimensional defect detection and three-dimensional defect detection can be achieved at the same time, thereby improving the comprehensiveness and accuracy of defect detection. On this basis, since the mapping relationship can achieve the correspondence between the position information in the transformed captured image and the position information in the point cloud image, the defect information of different defect types can be mapped to the same three-dimensional model based on the mapping relationship, thereby improving the reliability and accuracy of the defect detection results.

[0051] In one example, in an application scenario where the detection object is a sphere, the point cloud image may include a second position information set corresponding to the detection object obtained by using a point cloud acquisition device. The method of determining the first defect information based on the point cloud image may include: locating suspected defect areas, merging defects in multiple fields of view, and screening and measuring defect areas.

[0052] Locating suspected defect areas refers to processing the point cloud images of each field of view separately, dividing the point cloud images into blocks, and performing curve fitting and surface fitting on the divided point cloud blocks, so as to more accurately locate the suspected defect areas. Specifically, the point cloud image can be divided into multiple candidate point cloud blocks, and multiple candidate defect areas can be determined based on the multiple candidate point cloud blocks.

[0053] In one example, the method for obtaining candidate point cloud blocks can be configured according to actual business needs and is not limited here. For example, the method for obtaining candidate point cloud blocks may include at least one of the following: a voxel-based method, a clustering-based method, a deep learning-based method, and a segmentation-based method. The voxel-based method refers to dividing the point cloud image into regional blocks based on voxels of a fixed size. The clustering-based method refers to clustering according to the distance or density between points in the point cloud image, and aggregating adjacent points into a block. The deep learning-based method refers to using a convolutional neural network or a semantic segmentation network to segment the point cloud and generate a label for each point, thereby realizing the division of point cloud blocks. The segmentation-based method refers to using a segmentation method to divide the original point cloud into smaller subsets.

[0054] Multi-field defect merging refers to obtaining the stitching parameters of multiple candidate defect areas through a point cloud stitching algorithm, and stitching the multiple candidate defect areas using the stitching parameters to obtain multiple merged defect areas. Specifically, the stitching parameters can be determined based on the field of view of the point cloud image, and multiple candidate defect areas under multiple fields of view are processed based on the stitching parameters to obtain multiple merged defect areas.

[0055] Screening and measuring defect areas refers to screening multiple merged defect areas and calculating the sizes of the multiple screened defect areas to obtain a target defect area. Specifically, multiple merged defect areas can be screened based on screening rules to obtain at least one target defect area. In the application scenario where the detection object is a sphere, the target defect area can refer to a defect area with depth information on the detection object.

[0056] In one example, the screening rules can be configured according to actual business needs and are not limited here. For example, the screening rules can be used to distinguish different defect areas and merge the same defect areas. Distinguishing different defect areas means identifying the defects in the merged defect area as different defects. Merging the same defect area means identifying defects with similar characteristics or repeated defects as the same defect.

[0057] According to the embodiments of the present disclosure, by dividing the point cloud image into multiple candidate point cloud blocks and determining multiple candidate defect areas for each candidate point cloud block, the amount of data that needs to be processed in each part can be reduced, thereby improving the accuracy of extracting candidate defect areas. By processing the candidate defect areas under different fields of view separately, a merged defect area that fully grasps the defect area can be obtained, thereby improving the accuracy of subsequent detection. On this basis, by using screening rules to screen multiple merged defect areas, the accuracy of the first defect information can be improved, and accurate positioning and effective identification of defects can be achieved.

[0058] The above describes how to determine the first defect information based on the point cloud image. Figure 3 The method of obtaining the candidate defect area is further described.

[0059] Figure 3 An example schematic diagram of a process for obtaining candidate defect regions according to an embodiment of the present disclosure is schematically shown.

[0060] like Figure 3 As shown, in 300, the point cloud image may be divided to obtain multiple candidate point cloud blocks 301 to improve accuracy. Each candidate point cloud block 301 includes N rows of candidate point cloud data, for example, each candidate point cloud block 301 may include candidate point cloud data 301_1, ..., candidate point cloud data 301_N.

[0061] Taking the candidate point cloud data 301_1 in the candidate point cloud block 301 as an example, a circle fitting can be performed on each row of the candidate point cloud data 301_1 to obtain the second radius 302_1 and the center 303_1 of the candidate point cloud data 301_1. The distance between each point in the candidate point cloud data 301_1 and the center 303_1 and the difference 304_1 between the second radius 302_1 are determined respectively.

[0062] In one example, the least square method can be used to find the fitting circle that best fits the candidate point cloud data 301_1, and the second radius 302_1 and the center 303_1 can be determined using the fitted circle equation. The center 303_1 refers to the center point of the fitting circle, and the second radius 302_1 refers to the distance from the center point of the fitting circle to any point on the circumference.

[0063] After obtaining the difference 304_1 of each point in the candidate point cloud data 301_1, operation S310 may be performed. In operation S310, is the difference 304_1 less than the first threshold? If so, the points in the candidate point cloud data 301_1 whose difference 304_1 is less than the first threshold may be retained. By analogy, taking the candidate point cloud data 301_N in the candidate point cloud block 301 as an example, a circle fitting may be performed on each row of the candidate point cloud data 301_N to obtain the second radius 302_N and the center 303_N of the candidate point cloud data 301_N. The distance between each point in the candidate point cloud data 301_N and the center 303_N, and the difference 304_N between the second radius 302_N are determined respectively. After obtaining the difference 304_N of each point in the candidate point cloud data 301_N, operation S320 may be performed. In operation S320, is the difference 304_N less than the first threshold? If so, the points in the candidate point cloud data 301_N whose difference value 304_N is less than the first threshold value may be retained.

[0064] After determining the candidate point cloud data belonging to the normal area from the N candidate point cloud data, these candidate point cloud data may be merged to obtain merged point cloud data 305 .

[0065] In one example, the specific method of merging the candidate point cloud data can be configured according to actual business needs and is not limited here. For example, the merging method may include at least one of the following: an accumulation-based method, a feature matching-based method, a nearest neighbor iteration-based method, and a topological structure transformation-based method. The accumulation-based method refers to adding the candidate point cloud data. The feature matching-based method refers to using feature descriptors to extract features of different candidate point cloud data, and using these features for matching and fusion. The nearest neighbor iteration-based method refers to aligning different candidate point cloud data in the same coordinate system and optimizing the minimum distance between each point through iteration. The topological structure transformation-based method refers to connecting and merging different candidate point cloud data by establishing topological relationships between different candidate point cloud data.

[0066] After the merged point cloud data 305 is obtained, a sphere fitting may be performed on the merged point cloud data 305 to obtain a first radius 306 and a sphere center 307. A difference 308 between the distance from the merged point cloud data 305 to the sphere center 307 and the first radius 306 is determined.

[0067] In one example, the least square method can be used to find the fitting sphere that best fits the merged point cloud data 305, and determine the sphere center 307 and the first radius 306 of the fitting sphere. The sphere center 307 refers to the center point of the fitting sphere, that is, the point where the sum of the distances from all points on the fitted sphere surface to the center is the smallest. The first radius 306 refers to the distance from the center point of the fitting sphere to the point farthest from the sphere center.

[0068] For example, three points can be randomly selected from the merged point cloud data 305 as the initial fitting sphere center, and the following operations can be repeated on the initial fitting sphere center until the predetermined end condition is met, and the sphere center obtained when the predetermined end condition is met is determined as the sphere center 307 of the fitting sphere, and the distance from the point farthest from the sphere center 307 to the sphere center 307 is determined as the first radius 306. The predetermined end condition can be configured according to actual business needs and is not limited here. For example, the predetermined end condition can include one of the following: the number of iterations reaches the iteration number threshold, and the change amount of the sphere center is less than the change amount threshold.

[0069] For each point in the merged point cloud data 305, the distance from the point to the initial fitting sphere center can be calculated, and the point with the shortest distance is determined as the inner point. On this basis, the position of the initial fitting sphere center is re-estimated according to the inner point to obtain the updated initial fitting sphere center. For example, the average value of all the inner points can be used as the updated initial fitting sphere center. The initial radius is re-estimated according to the inner point to obtain the updated first radius. For example, the average value of the distances from all the inner points to the updated initial fitting sphere center can be used as the updated first radius.

[0070] After obtaining the difference 308, operation S330 can be performed. In operation S330, is the difference 308 greater than the second threshold? If not, the area corresponding to the merged point cloud data 305 can be determined as the normal area 309. If so, the area corresponding to the merged point cloud data 305 can be determined as the candidate defect area 310. In the application scenario where the detection object is a sphere, the candidate defect area 310 may refer to a candidate defect area with depth information on the detection object. The first threshold and the second threshold can be configured according to actual business needs and are not limited here. For example, the first threshold may be less than the second threshold, the first threshold may be 0.3, and the second threshold may be 0.6.

[0071] According to the embodiments of the present disclosure, by analyzing and merging point cloud data using circle fitting and sphere fitting methods and comparing the fitting data according to a preset threshold, candidate defect areas can be quickly identified while ensuring accuracy, which is beneficial to improving the accuracy and efficiency of subsequent defect detection.

[0072] In another example, each candidate point cloud block may include points of a first number of rows*a second number of columns.

[0073] For each of the first number of rows, a second number of points corresponding to the row may be fitted to obtain a first fitting value; a first difference between the first fitting value and the second number of points is determined. On this basis, a circle fitting is performed on the transverse curve based on the reconstruction principle to obtain a third radius and a first center of the circle; and a second difference between the distance between the second number of points and the first center of the circle and the third radius is determined.

[0074] For each column in the second number of columns, the first number of points corresponding to the column can be fitted to obtain a second fitting value; a third difference between the second fitting value and the first number of points is determined. On this basis, a circle fitting is performed on the longitudinal curve based on the reconstruction principle to obtain a fourth radius and a second center; a fourth difference between the distance between the first number of points and the second center and the fourth radius is determined.

[0075] On this basis, the smallest value among the first difference, the second difference, the third difference and the fourth difference can be determined as the target difference, and whether the target difference is greater than the fifth threshold can be determined. If not, the area corresponding to the target difference less than or equal to the fifth threshold can be determined as the normal area 309. If so, the area corresponding to the target difference greater than the fifth threshold can be determined as the candidate defect area 310. In the application scenario where the detection object is a sphere, the candidate defect area 310 can refer to the candidate defect area with depth information on the detection object.

[0076] The above describes how to obtain candidate defect areas. Figure 4 The method of obtaining the first defect information is further explained.

[0077] Figure 4 The diagram schematically shows an example of a process of screening multiple merged defect regions based on screening rules to obtain first defect information according to an embodiment of the present disclosure.

[0078] like Figure 4 As shown, in 400 , after obtaining the candidate defect regions 401 , multiple candidate defect regions 401 under the field of view may be transformed based on an angle 402 corresponding to the field of view to obtain multiple transformed defect regions 403 .

[0079] In one example, the angle corresponding to the field of view may include the angle of the image acquisition device for taking images relative to the detection object and the angle of the point cloud acquisition device for point cloud images relative to the detection object. The angle of the image acquisition device relative to the detection object refers to the angle difference between the image acquisition device and the detection object, which is used to indicate the position or direction of the image acquisition device relative to the detection object. The angle of the point cloud acquisition device relative to the detection object refers to the angle difference between the point cloud acquisition device and the detection object, which is used to indicate the position or direction of the point cloud acquisition device relative to the detection object.

[0080] After obtaining the transformed defect area 403 under multiple fields of view, the rotation matrix 404 can be determined based on the transformed defect area under the field of view and the transformed defect area under other fields of view. The rotation matrix 404 may include rotation parameters and translation parameters. The rotation parameters can be used to control the rotation angle of the transformed defect area 403 in the three-dimensional space. The translation parameters can be used to control the translation amount of the transformed defect area 403 at each three-dimensional coordinate in the three-dimensional space. The determination method of the rotation matrix 404 can be configured according to actual business needs and is not limited here. For example, the rotation matrix 404 can be determined based on the rotation vector in the three-dimensional space. Alternatively, the rotation matrix 404 can be determined based on the rotation parameters.

[0081] Based on the transformed defect area 403 and the rotation matrix 404 under the field of view, the transformed defect area 403 under each field of view is spliced ​​to obtain a merged defect area 405, so as to avoid dividing the same defect area into multiple defect areas, and merge the defects belonging to the same defect area under multiple fields of view, so as to ensure the detection accuracy.

[0082] According to an embodiment of the present disclosure, by considering multiple transformed defect areas under different fields of view, since different fields of view may expose different defects or defects in different parts, it is possible to comprehensively utilize the information of the image acquisition device and the point cloud acquisition device to improve the accuracy of merging defect areas.

[0083] After the merged defect region 405 is obtained, since the distances between the points inside different defect regions are usually far apart, the merged defect region 405 can be clustered to obtain a plurality of first sub-defect regions 406 with the goal of distinguishing different defect regions in the merged defect region 405. Distinguishing different defect regions in the merged defect region 405 means identifying and distinguishing different defect regions in the merged defect region 405.

[0084] In one example, clustering for the purpose of splitting refers to dividing multiple points in the merged defect area 405 according to their feature similarities, thereby forming different first sub-defect areas 406, so that the points in each first sub-defect area 406 are as similar as possible, and the points between different first sub-defect areas 406 are as dissimilar as possible. The clustering method for the purpose of splitting can be configured according to actual business needs and is not limited here. For example, the clustering method for the purpose of splitting can include at least one of the following: K-means clustering method, hierarchical clustering method and density clustering method, etc.

[0085] After obtaining a plurality of first sub-defect regions 406, the maximum depth information in each first sub-defect region 406 may be determined, and operation S410 may be performed. The method for determining the maximum depth information may be configured according to actual business requirements and is not limited herein. For example, the depth values ​​of the points inside the first sub-defect region 406 may be determined, and the depth value with the largest value may be determined as the maximum depth information.

[0086] In operation S410, is it determined whether the maximum depth information is greater than the third threshold? If not, the first defect sub-region 406 can be deleted. If yes, since the distances between the points inside the same defect region are usually close, the first defect sub-region 406 can be clustered with the goal of merging the same defect regions in the first defect sub-region 406 to obtain a plurality of second defect sub-regions 407. Merging the same defect regions in the first defect sub-region 406 means clustering the same or similar defect regions in the first defect sub-regions together to form second defect sub-regions with common characteristics.

[0087] In one example, clustering for the purpose of merging refers to gradually merging points in different first sub-defect regions 406 until a number of second sub-defect regions 407 are formed, so that the points in each second sub-defect region 407 are as similar as possible, and the similarity between the first sub-defect region 406 and the second sub-defect region 407 is low. The clustering method for the purpose of merging can be configured according to actual business needs and is not limited here. For example, the clustering method for the purpose of merging can include at least one of the following: a hierarchical clustering method and a density clustering method, etc.

[0088] After obtaining a plurality of second sub-defect regions 407, a maximum distance 408 within the second sub-defect region 407 may be determined, and operation S420 may be performed. The maximum distance 408 refers to the distance between the farthest position information in a preset direction. The maximum distance 408 may be configured according to actual business needs and is not limited here. For example, the size information of each second sub-defect region 407 may be calculated using a minimum bounding box method, and the maximum distance 408 may be determined based on the size information.

[0089] In operation S420, is it determined whether the maximum distance 408 is greater than a fourth threshold? If not, the second sub-defect area 407 may be deleted. If yes, the second sub-defect area 407 may be determined as a target defect area 409. In an application scenario where the detection object is a sphere, the target defect area 409 may refer to a defect area with depth information on the detection object.

[0090] According to an embodiment of the present disclosure, by adopting a multi-level clustering method with the goal of distinguishing different defect areas in the merged defect area and with the goal of merging the same defect areas in the first sub-defect area, and at the same time using the set third threshold and fourth threshold judgment conditions, the target defect area can be identified and located more accurately, thereby improving the accuracy and efficiency of defect detection.

[0091] The above describes how to obtain the first defect information. Figure 5 The method of obtaining the maximum distance is further described.

[0092] Figure 5 An example schematic diagram of a maximum distance determination process according to an embodiment of the present disclosure is schematically shown.

[0093] like Figure 5 As shown, in 500, the point cloud image 501 can be processed to obtain a binary image 502. The binary image 502 may refer to an image that includes only two attribute values, one of which represents the existence of a defect area and the other represents the existence of a background area. The method for obtaining the binary image 502 can be configured according to actual business needs and is not limited here. For example, the method for obtaining the binary image 502 may include at least one of the following: a method based on threshold segmentation, a method based on edge detection, and a method based on morphological operations, etc.

[0094] The method based on threshold segmentation refers to converting the point cloud image 501 into a binary image 502 by setting a threshold, that is, points exceeding the threshold are set to white (255), and points below the threshold are set to black (0). The method based on edge detection refers to extracting edge information in the point cloud image 501 using an edge detection algorithm, and generating a binary image 502 based on the edge information. The method based on morphological operations refers to processing the point cloud image 501 through morphological operations such as dilation, erosion, opening operation, and closing operation to obtain a binary image 502.

[0095] In one example, the attribute value of the position corresponding to the candidate defect area in the point cloud image 501 can be set to a first value, and the attribute value of the position other than the candidate defect area in the point cloud image 501 can be set to a second value to obtain a binary image 502. In another example, the attribute value of the position corresponding to the second sub-defect area in the point cloud image 501 can be set to a first value, and the attribute value of the position other than the second sub-defect area in the point cloud image 501 can be set to a second value to obtain a binary image 502. The first value and the second value can be configured according to actual business needs, and are not limited here. For example, the first value can be 1, and the second value can be 0. After obtaining the binary image 502, the binary image 502 can be traversed along a preset direction based on the size of the binary image 502 in the preset direction to obtain a boundary point set 505. The preset direction can be configured according to actual business needs, and are not limited here. For example, the preset direction can include a horizontal direction and a vertical direction, the horizontal direction can correspond to a row (i.e., i), and the vertical direction can correspond to a column (i.e., j). The boundary point set 505 may include a plurality of boundary point pairs, and the boundary point pairs may represent the boundary position between the foreground and the background. The binary image 502 may include M rows*N columns of position information.

[0096] For example, for each row or column in the binary image 502, starting from the first point of each row or column, based on a predetermined condition, the data of each row and column in the binary image 502 can be traversed in a loop along a preset direction to detect changes in attribute values ​​and implement boundary point retrieval. The predetermined condition can be configured according to actual business needs and is not limited here. For example, the predetermined condition can be that the position information is the last position information of the row or column. Thus, when it is found that the attribute value transitions from the background color to the foreground color, the point can be recorded as a boundary point; when it is found that the attribute value transitions from the foreground color to the background color, the point can be recorded as a boundary point; two boundary points in the same row or column can be used as a boundary point pair.

[0097] According to the embodiments of the present disclosure, by binarizing the point cloud image, the position of the second sub-defect area in the entire point cloud image can be emphasized and highlighted. By traversing the binary image along a preset direction to obtain a boundary point set, and taking the distance between the farthest pair of boundary points in the boundary point set as the maximum distance, it can provide a basis for the subsequent determination of the target defect area, thereby improving the accuracy of the first defect information.

[0098] In one example, taking the preset direction as the horizontal direction as an example, the process of traversing the binary image 502 along the horizontal direction based on the size of the binary image 502 in the horizontal direction to obtain the boundary point set 505 is described.

[0099] First, set the flag to 0, and start from the point where the position information is i=0, j=0, and iterate the i++ operation until i=M-1.

[0100] During the iterative operation, if the pixel value of a certain position information is not 0, the point can be determined as the first boundary point 503, and the flag bit flag is set to 1. The j++ operation is performed in the row to continue traversing until it is found that the pixel value of a certain position information is 0. If the position information is not the last position information, the previous position information of the position information is determined as the second boundary point 504, and the flag bit flag is set to 0. If the position information is the last position information, the position information is determined as the second boundary point 504, and the flag bit flag is set to 0.

[0101] In operation S510, is it determined whether the pixel value of the position information (i, j) is 0? If yes, operation S520 may be performed. If no, operation S530 may be performed.

[0102] In operation S520, is it determined whether flag is equal to 0? If so, the flag may be set to 1, and the position information (i, j) may be determined as the first boundary point 503. If not, operation S540 may be performed.

[0103] In operation S530, is it determined whether flag is equal to 1? If so, the flag may be set to 0, and the position information (i, j-1) may be determined as the first boundary point 503. If not, operation S540 may be performed.

[0104] In operation S540, is it determined whether j is less than N-1? If not, S540 may be executed. If yes, an operation of j++ may be executed, and S510 to S540 may be repeatedly executed based on the result of j++.

[0105] In operation S550, is it determined whether flag is equal to 1? If so, the position information (i, j) may be determined as the second boundary point 504. If not, S560 may be executed.

[0106] In operation S560, is it determined whether i is less than M-1? If yes, the operation of i++ can be performed, and S510-S560 can be repeatedly performed based on the result of i++. If no, the first boundary point 503 and the second boundary point 504 can be stored as a boundary point pair in the boundary point set 505.

[0107] In another example, taking the preset direction as the vertical direction as an example, the process of traversing the binary image 502 along the vertical direction based on the size of the binary image 502 in the vertical direction to obtain the boundary point set 505 is described.

[0108] First, set the flag to 0, and start from the point where the position information is i=0, j=0, and iterate the j++ operation until j=N-1.

[0109] During the iterative operation, if the pixel value of a certain position information is not 0, the point can be determined as the first boundary point 503, and the flag bit flag is set to 1. The i++ operation is performed in the column to continue traversing until it is found that the pixel value of a certain position information is 0. If the position information is not the last position information, the previous position information of the position information is determined as the second boundary point 504, and the flag bit flag is set to 0. If the position information is the last position information, the position information is determined as the second boundary point 504, and the flag bit flag is set to 0.

[0110] In operation S510, is it determined whether the pixel value of the position information (i, j) is 0? If yes, operation S520 may be performed. If no, operation S530 may be performed.

[0111] In operation S520, is it determined whether flag is equal to 0? If so, the flag may be set to 1, and the position information (i, j) may be determined as the first boundary point 503. If not, operation S570 may be performed.

[0112] In operation S530, is it determined whether flag is equal to 1? If so, flag can be set to 0, and the position information (i-1, j) can be determined as the first boundary point 503. If not, operation S570 can be performed.

[0113] In operation S570, is it determined whether i is less than M-1? If not, S550 may be executed. If yes, an operation of i++ may be executed, and S510 to S570 may be repeatedly executed based on the result of i++.

[0114] In operation S550, is it determined whether flag is equal to 1? If so, the position information (i, j) may be determined as the second boundary point 504. If not, S580 may be executed.

[0115] In operation S580, is it determined whether j is less than N-1? If so, the operation of j++ can be performed, and S510-S580 can be repeatedly performed based on the result of j++. If not, the first boundary point 503 and the second boundary point 504 can be stored as a boundary point pair in the boundary point set 505.

[0116] According to an embodiment of the present disclosure, by determining the position information according to specific conditions and respectively determining the first boundary point and the second boundary point when encountering position information that meets the conditions, the boundary point pair can be accurately determined, thereby improving the accuracy of the subsequent first defect information.

[0117] After obtaining the boundary point set 505, the maximum distance 506 can be determined according to the boundary point set 505. The maximum distance 506 refers to the farthest distance between a pair of boundary points in the boundary point set 505. The method for determining the maximum distance 506 can be configured according to actual business needs and is not limited here. For example, the distance between a pair of boundary points that are farthest apart from each other among the plurality of boundary point pairs can be used as the maximum distance 506.

[0118] In one example, after obtaining the boundary point set 505, the boundary point set 505 may be downsampled to obtain a predetermined number of boundary point pairs, and then the distance between a pair of boundary points that are farthest apart among the predetermined number of boundary point pairs is used as the maximum distance 506. In this way, the order of magnitude of the boundary point pairs can be reduced, and the processing efficiency can be improved.

[0119] The above describes how to determine the first defect information based on the point cloud image. The following describes how to determine the second defect information based on the captured image.

[0120] In an application scenario where the detection object is a sphere, the captured image may include a first position information set corresponding to the detection object obtained by using an image acquisition device. After obtaining the captured image, the second defect information may be determined based on a transformed captured image obtained by processing the captured image.

[0121] In one example, the captured image can be transformed according to the brightness image corresponding to the point cloud image to obtain a transformed captured image, so as to align the brightness image and the captured image in the height direction and the width direction respectively. The transformation process includes at least one of the following: scaling transformation, flipping transformation and image stitching transformation.

[0122] For example, scaling transformation refers to adjusting the size of the captured image, such as increasing or decreasing the size of the captured image while keeping the aspect ratio of the captured image unchanged. In one example, scaling transformation of the captured image can be implemented using a first preset function, and the first preset function can be a resize function.

[0123] Alternatively, the flip transformation refers to mirror-flipping the captured image along a horizontal axis or a vertical axis, for example, horizontal flipping with the vertical center line of the captured image as the axis of symmetry, and vertical flipping with the horizontal center line of the captured image as the axis of symmetry. In one example, the flip transformation of the captured image can be implemented using a second preset function, and the second preset function can be a flip function.

[0124] Alternatively, the image stitching transformation refers to cutting and stitching the captured images according to a predetermined rule. In one example, the image stitching transformation of the captured images can be implemented using a third preset function, and the third preset function can be an hconcat function and a vconcat function.

[0125] In another example, after obtaining the transformed captured image, target detection can be performed on the transformed captured image to obtain second defect information. The specific target detection method can be configured according to actual business needs and is not limited here. The second defect information may include bounding boxes of multiple candidate defect areas. The bounding box can be used to characterize the location of each candidate defect area. In an application scenario where the detection object is a sphere, the second defect information can be used to indicate defects located on the surface of the detection object.

[0126] According to the embodiments of the present disclosure, by transforming the captured image according to the brightness image of the point cloud image, a transformed captured image can be obtained, which helps to improve the quality of the captured image and can also achieve alignment between the captured image and the brightness image. On this basis, by performing target detection on the transformed captured image, the defective part in the captured image can be effectively identified, thereby improving the accuracy of the second defect information.

[0127] In the embodiment of the present disclosure, the second defect information can be obtained by inputting the transformed captured image into the target detection model. Figure 6 Describe the process of training an object detection model.

[0128] Figure 6 An example schematic diagram of a target detection model training process according to an embodiment of the present disclosure is schematically shown.

[0129] like Figure 6 As shown, in 600, the originally acquired sample shot image 601 may be transformed to obtain a transformed sample shot image 602. The transformed sample shot image 602 is an image aligned with the brightness image of the sample point cloud image. The transformation process may include at least one of the following: scaling transformation, flipping transformation, and image splicing transformation.

[0130] After obtaining the transformed sample shot image 602, the sample defect areas in the transformed sample shot image 602 can be annotated to obtain respective bounding boxes 603 of the sample defect areas. The annotation method and file type can be configured according to actual business needs and are not limited here. For example, the file type can be a JSON (JavaScript Object Notation) file.

[0131] The labeling method may include at least one of the following: a manual labeling method, a machine learning-based labeling method, and a detection-based labeling method. The manual labeling method refers to manually drawing a rectangle or polygon to label the bounding box of the target object. The machine learning-based labeling method refers to using computer vision or deep learning technology to pre-train the model to achieve automatic detection. The detection-based labeling method refers to using a segmentation model to perform pixel-level recognition and segmentation.

[0132] After obtaining the bounding boxes 603 of the sample defect regions, the JSON file can be converted into a TXT (Text File) file used for model training, and the data can be organized into a training sample set 604. In addition, the data can be divided into a training sample set 604, a verification sample set, and a test sample set according to a predetermined ratio.

[0133] After obtaining the training sample set 604, the deep learning model 605 can be trained using the training sample set 604 to obtain the target detection model, and the model file corresponding to the target detection model can be exported as a general type model file 606 for subsequent use without retraining the model. For example, the general type can be ONNX (Open Neural Network Exchange). The deep learning model 605 can be configured according to actual business needs, which is not limited here.

[0134] The above describes the training process of the target detection model. Figure 7 How to use the target detection model to process the transformed captured image to obtain the second defect information, and how to obtain the defect detection result are further explained.

[0135] Figure 7 An example schematic diagram of a target detection model application process according to an embodiment of the present disclosure is schematically shown.

[0136] like Figure 7 As shown, in 700, the captured image 701 can be transformed according to the brightness image corresponding to the point cloud image to obtain the transformed captured image 702, so as to align the brightness image and the captured image 701 in the height direction and the width direction respectively. The transformation process includes at least one of the following: scaling transformation, flipping transformation and image stitching transformation.

[0137] The general type model file 703 is converted into a predetermined type target detection model 704. The predetermined type refers to an optimized and serialized deep learning model. For example, the predetermined type can be a TensorRT engine model to improve the reasoning speed. The transformed captured image 702 is input into the target detection model 704 to obtain the bounding boxes 705 of the candidate defect areas. In the application scenario where the detection object is a sphere, the bounding boxes 705 of the candidate defect areas can be used to indicate defects located on the surface of the detection object.

[0138] According to the first position information in the transformed captured image 702 and the second position information in the brightness image of the point cloud image, a mapping relationship 706 is determined. The mapping relationship 706 can be used to map any coordinate on the transformed captured image 702 to a corresponding position on the brightness image of the point cloud image.

[0139] Based on the mapping relationship 706, the bounding boxes 705 of the plurality of candidate defect regions are mapped to the brightness image of the point cloud image, and the defect information 707 in the spherical coordinate system is obtained by rotation and splicing. On this basis, the first defect information 708 and the defect information 707 can be fused to determine the defect detection result 709. The specific fusion method can be configured according to actual business needs and is not limited here. For example, the fusion method may include at least one of the following: a fusion method based on a deep learning model, a fusion method based on feature extraction, and a fusion method based on projection transformation, etc.

[0140] The fusion method based on deep learning model refers to fusing the two-dimensional position information into the three-dimensional point cloud by using deep learning models such as convolutional neural network or recursive neural network. The fusion method based on feature extraction refers to extracting features from the first defect information 708 and the defect information 707 respectively, and then fusing these features. The fusion method based on projection transformation refers to projecting the two-dimensional position information into three-dimensional space and aligning it with the original point cloud data.

[0141] According to the embodiments of the present disclosure, since the mapping relationship is determined based on the first position information in the transformed captured image and the second position information in the brightness image of the point cloud image, by establishing a spatial mapping between the two, it is possible to link information in different expressions, thereby achieving subsequent defect information fusion. On this basis, by combining the second defect information obtained in the captured image with the first defect information obtained in the point cloud image based on the mapping relationship, the accuracy of the defect detection result can be improved.

[0142] Figure 8 A block diagram of a defect detection device according to an embodiment of the present disclosure is schematically shown.

[0143] like Figure 8As shown, the defect detection device 800 may include an acquisition module 810 , a first determination module 820 , a second determination module 830 and a third determination module 840 .

[0144] The acquisition module 810 is used to acquire the captured images and point cloud images of the detection object in multiple fields of view.

[0145] The first determination module 820 is used to determine first defect information based on at least one candidate defect area obtained by processing the point cloud image.

[0146] The second determination module 830 is used to determine second defect information based on the transformed captured image obtained by processing the captured image, where the first defect information and the second defect information are for different defect types.

[0147] The third determination module 840 is used to determine the defect detection result based on the relationship between the transformed captured image and the point cloud image and according to the first defect information and the second defect information.

[0148] According to the embodiments of the present invention, any one or more of the modules, or at least part of the functions of any one of them, can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0149] It should be noted that the defect detection device part in the embodiment of the present disclosure corresponds to the defect detection method part in the embodiment of the present disclosure. The description of the defect detection device part specifically refers to the defect detection method part, which will not be repeated here.

[0150] Fig. 9 A block diagram of an electronic device suitable for implementing a defect detection method according to an embodiment of the present disclosure is schematically shown. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0151] like Fig. 9As shown, the computer electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0152] In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904.

[0153] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0154] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the defect detection method according to the embodiment of the present disclosure is implemented.

[0155] In the present disclosure, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0156] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the defect detection method provided by the embodiment of the present disclosure.

[0157] When the computer program is executed by the processor 901, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0158] According to the embodiments of the present disclosure, program codes for executing the computer programs provided by the embodiments of the present disclosure may be written in any combination of one or more programming languages.

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. It should also be noted that in some alternative implementations, the functions marked in the blocks may also occur in an order different from that marked in the accompanying drawings.

[0160] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A defect detection method, comprising: Obtaining photographic images and point cloud images of the detection object in multiple fields of view; Processing the point cloud image to obtain first defect information; determining second defect information based on a transformed captured image obtained by processing the captured image, wherein the first defect information and the second defect information are for different defect types; as well as Based on a mapping relationship between respective position information of the transformed captured image and the point cloud image, a defect detection result is determined according to the first defect information and the second defect information.

2. The method according to claim 1, wherein: The processing of the point cloud image to obtain the first defect information includes: Based on the point cloud image, determining a plurality of candidate defect areas; Based on the field of view, processing a plurality of candidate defect regions under the plurality of fields of view to obtain a plurality of merged defect regions; and The plurality of merged defect regions are screened based on a screening rule to obtain the first defect information, wherein the screening rule is used to distinguish different defect regions and merge the same defect regions.

3. The method according to claim 2, wherein: The point cloud image includes a plurality of candidate point cloud data; The determining of the plurality of candidate defect areas based on the point cloud image comprises: Fitting each of the candidate point cloud data to obtain fitting data; In response to the candidate point cloud data and the fitting data satisfying a preset condition, merging the candidate point cloud data to obtain merged point cloud data; Performing spherical fitting on each of the merged point cloud data to obtain a first radius and a spherical center; and In response to a difference between a distance from the combined point cloud data to the sphere center and the first radius being greater than a first threshold, a region corresponding to the combined point cloud data is determined as the candidate defect region.

4. The method according to claim 3, wherein: The fitting of each candidate point cloud data to obtain the fitting data comprises: Performing circle fitting on each candidate point cloud data to obtain a second radius and a center of each candidate point cloud data; The preset condition includes: a difference between the distance between the candidate point cloud data and the center of the circle and the second radius is less than a second threshold, and the second threshold is less than the first threshold.

5. The method according to claim 2, wherein: The processing of the plurality of candidate defect regions under the plurality of fields of view based on the field of view to obtain the plurality of merged defect regions comprises: transforming a plurality of candidate defect regions under the field of view based on an angle corresponding to the field of view to obtain a plurality of transformed defect regions, wherein the angle corresponding to the field of view includes an angle of an image acquisition device for capturing the image relative to the inspection object and an angle of a point cloud acquisition device for the point cloud image relative to the inspection object; and Based on the rotation matrix determined by the transformed defect region under the field of view and the transformed defect region under other fields of view, the transformed defect region under each field of view is spliced ​​to obtain the merged defect region.

6. The method according to claim 2, wherein: The first defect information includes at least one target defect area; The screening of the plurality of merged defect regions based on the screening rule to obtain the first defect information includes: With the goal of distinguishing different defect regions in the merged defect region, clustering each of the merged defect regions to obtain a plurality of first sub-defect regions; In response to the maximum depth information in the first defect sub-region being greater than a third threshold, clustering the first defect sub-regions with the goal of merging identical defect regions in the first defect sub-regions to obtain a plurality of second defect sub-regions; and In response to the maximum distance between the farthest distances of the position information in the second defect sub-region in the preset direction being greater than a fourth threshold, the second defect sub-region is determined as the target defect region.

7. The method according to claim 6, wherein: The maximum distance is determined as follows: Along the preset direction, based on the size of the binary image of the point cloud image in the preset direction, traverse the binary image to obtain a boundary point set, wherein the boundary point set includes a plurality of boundary point pairs; as well as The distance between a pair of boundary points that are farthest apart from each other among the plurality of boundary point pairs is taken as the maximum distance.

8. The method according to claim 7, wherein: The binary image is determined by one of the following methods: Determining the binary image according to the candidate defect area; or The attribute value of the position corresponding to the second sub-defect area in the point cloud image is set to a first value, and the attribute value of the position other than the second sub-defect area is set to a second value, so as to obtain the binary image.

9. The method according to claim 7, wherein: The binary image includes M rows*N columns of position information, where M and N are both positive integers; The step of traversing the binary image along a preset direction based on a size of the binary image in the preset direction to obtain a set of boundary points includes: For each row in M ​​or each column in N, In response to the attribute value corresponding to the position information being the first value, using the position information as a first boundary point; In response to the attribute value corresponding to the position information being the second value, determining a second boundary point based on the position information, wherein, if the position information does not satisfy a predetermined condition, using previous position information adjacent to the position information as the second boundary point; and if the position information satisfies the predetermined condition, using the position information as the second boundary point; and The first boundary point and the second boundary point are taken as the boundary point pair.

10. The method according to any one of claims 1 to 9, wherein: The determining of the second defect information based on the transformed captured image obtained by processing the captured image comprises: According to the brightness image of the point cloud image, transform the captured image to obtain the transformed captured image, wherein the transform process includes at least one of the following: scaling transform, flipping transform and image stitching transform; and The transformed captured image is input into a target detection model to obtain the second defect information, where the second defect information includes respective bounding boxes of a plurality of candidate defect regions.

11. The method according to claim 10, wherein: The determining of the defect detection result based on the mapping relationship between the position information of the transformed captured image and the point cloud image, according to the first defect information and the second defect information, comprises: determining the mapping relationship according to the first position information in the transformed captured image and the second position information in the brightness image of the point cloud image; and Based on the mapping relationship, the defect detection result is determined according to the first defect information and defect information obtained by mapping the respective bounding boxes of the plurality of candidate defect regions to the point cloud image.

12. The method according to claim 10, wherein: The target detection model is trained in the following way: Performing transformation processing on the plurality of sample captured images respectively to obtain a plurality of transformed sample captured images; Marking the sample defect regions in the plurality of transformed sample captured images respectively to obtain respective bounding boxes of the plurality of sample defect regions; generating a training sample set according to the plurality of sample captured images and respective bounding boxes of the plurality of sample defect regions; as well as The deep learning model is trained using the training sample set to obtain the target detection model.

13. The method according to claim 1, wherein: The inspection object is a sphere, the first defect information is for defects with depth information, and the second defect information is for defects located on the surface of the inspection object.

14. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images