Foreign matter detection method, device and equipment for to-be-tested area and storage medium

Abstract

The present disclosure relates to a method and device for detecting foreign matter in a to-be-detected area and a storage medium. The method comprises: collecting three-dimensional point cloud data of the to-be-detected area; determining two-dimensional image data of the to-be-detected area based on the three-dimensional point cloud data; detecting foreign matter in the to-be-detected area based on the three-dimensional point cloud data to obtain a three-dimensional detection result, and detecting foreign matter in the to-be-detected area based on the two-dimensional image data to obtain a two-dimensional detection result; and determining a foreign matter detection result of the to-be-detected area based on the three-dimensional detection result and the two-dimensional detection result. According to the present disclosure, three-dimensional detection and two-dimensional detection are performed based on the collected three-dimensional point cloud data of the to-be-detected area, without the need to separately collect two-dimensional data, thereby reducing detection costs and improving foreign matter detection efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of foreign object detection, and in particular to methods, apparatus, equipment and storage media for detecting foreign objects in a region to be tested. Background Technology

[0002] During the assembly of terminal devices, the presence of foreign objects in the assembly area can lead to assembly failure, damage to components, and even injury to users. Therefore, the importance of pre-assembling foreign objects in the assembly area is self-evident, and ensuring zero omissions in foreign object detection is a pressing issue that the electronics industry needs to address.

[0003] In related technologies, the methods of detecting foreign objects in the area to be tested, such as using hardware mechanisms or manual visual inspection, suffer from problems such as high false negative rates and low detection efficiency. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a method, apparatus, device and storage medium for detecting foreign objects in the area to be tested.

[0005] According to a first aspect of the present disclosure, a method for detecting foreign objects in a test area is provided, comprising: acquiring three-dimensional point cloud data of the test area; determining two-dimensional image data of the test area based on the three-dimensional point cloud data; performing foreign object detection on the test area based on the three-dimensional point cloud data to obtain a three-dimensional detection result, and performing foreign object detection on the test area based on the two-dimensional image data to obtain a two-dimensional detection result; and determining a foreign object detection result of the test area based on the three-dimensional detection result and the two-dimensional detection result.

[0006] In one embodiment, determining the foreign object detection result of the area to be tested based on the three-dimensional detection result and the two-dimensional detection result includes: merging the foreign object data points in the three-dimensional detection result and the foreign object data points in the two-dimensional detection result, and determining the merged foreign object data points as the foreign object detection result of the area to be tested; or performing a verification detection on the three-dimensional detection result or the two-dimensional detection result, and determining the verification detection result as the foreign object detection result of the area to be tested.

[0007] In one embodiment, merging the foreign object data points in the three-dimensional detection result and the foreign object data points in the two-dimensional detection result includes: converting the foreign object data points in the three-dimensional detection result into two-dimensional foreign object data points, and taking the union of the two-dimensional foreign object data points and the foreign object data points in the two-dimensional detection result as the merged foreign object data points; or converting the foreign object data points in the two-dimensional detection result into three-dimensional foreign object data points, and taking the union of the three-dimensional foreign object data points and the foreign object data points in the three-dimensional detection result as the merged foreign object data points.

[0008] In one embodiment, the step of verifying the three-dimensional detection result or the two-dimensional detection result includes: re-examining the three-dimensional detection result to obtain a first verification detection result, and determining the first verification detection result as the foreign object detection result of the area to be tested; or, re-examining the two-dimensional detection result to obtain a second verification detection result, and determining the second verification detection result as the foreign object detection result of the area to be tested.

[0009] In one embodiment, re-inspecting the three-dimensional detection result to obtain a first verification detection result includes: converting foreign object data points in the two-dimensional detection result into three-dimensional foreign object data points; determining three-dimensional re-inspection foreign object data points, wherein the three-dimensional re-inspection foreign object data points are foreign object data points in the three-dimensional foreign object data points that are different from the foreign object data points in the three-dimensional detection result; re-inspecting the three-dimensional re-inspection foreign object data points to obtain a three-dimensional re-inspection result; and merging the three-dimensional re-inspection result and the three-dimensional detection result to obtain the first verification detection result.

[0010] In one embodiment, the method further includes: in response to the presence of foreign object data points in the three-dimensional re-inspection result, updating a first detection parameter to a second detection parameter; the first detection parameter is a three-dimensional detection parameter used when detecting foreign objects in the area to be tested based on the three-dimensional point cloud data, and the second detection parameter is a detection parameter used when re-inspecting the three-dimensional re-inspection foreign object data points; the parameter value corresponding to the first detection parameter is greater than the parameter value corresponding to the second detection parameter. Based on the first verification detection result, the detection parameters used for three-dimensional detection are adjusted.

[0011] In one embodiment, the two-dimensional detection result includes a two-dimensional AI detection result and a two-dimensional edge detection result. The step of re-examining the two-dimensional detection result to obtain a second verification detection result includes: converting foreign object data points in the three-dimensional detection result into two-dimensional foreign object data points; determining two-dimensional re-examining foreign object data points, wherein the two-dimensional re-examining foreign object data points are foreign object data points in the two-dimensional foreign object data points that differ from those in the two-dimensional AI detection result, and foreign object data points in the two-dimensional edge detection result that differ from those in the two-dimensional AI detection result; re-examining the two-dimensional re-examining data points based on an unsupervised detection network to determine a two-dimensional AI re-examining result; and merging the two-dimensional AI re-examining result and the two-dimensional AI detection result to obtain the second verification detection result.

[0012] In one embodiment, the method further includes: determining foreign object detection points for two-dimensional AI detection from the two-dimensional re-inspection results, and adding the foreign object detection points to the two-dimensional AI training samples, wherein the two-dimensional AI training samples are used to train a two-dimensional AI detection model, and the two-dimensional AI detection model is used to determine the two-dimensional AI detection results.

[0013] According to a second aspect of the present disclosure, a foreign object detection device for a test area is provided, comprising: a data acquisition unit for acquiring three-dimensional point cloud data of the test area; a conversion unit for determining two-dimensional image data of the test area based on the three-dimensional point cloud data; a determination unit for performing foreign object detection on the test area based on the three-dimensional point cloud data to obtain a three-dimensional detection result, and performing foreign object detection on the test area based on the two-dimensional image data to obtain a two-dimensional detection result; and a processing unit for determining the foreign object detection result of the test area based on the three-dimensional detection result and the two-dimensional detection result.

[0014] In one embodiment, the processing unit determines the foreign object detection result of the area to be tested based on the three-dimensional detection result and the two-dimensional detection result in the following manner: merging the foreign object data points in the three-dimensional detection result and the foreign object data points in the two-dimensional detection result, and determining the merged foreign object data points as the foreign object detection result of the area to be tested; or performing a verification detection on the three-dimensional detection result or the two-dimensional detection result, and determining the verification detection result as the foreign object detection result of the area to be tested.

[0015] In one embodiment, the processing unit merges the foreign object data points in the three-dimensional detection result and the foreign object data points in the two-dimensional detection result in the following manner: converting the foreign object data points in the three-dimensional detection result into two-dimensional foreign object data points, and taking the union of the two-dimensional foreign object data points and the foreign object data points in the two-dimensional detection result as the merged foreign object data points; or converting the foreign object data points in the two-dimensional detection result into three-dimensional foreign object data points, and taking the union of the three-dimensional foreign object data points and the foreign object data points in the three-dimensional detection result as the merged foreign object data points.

[0016] In one embodiment, the processing unit performs a verification test on the three-dimensional detection result or the two-dimensional detection result in the following manner: the three-dimensional detection result is re-examined to obtain a first verification test result, and the first verification test result is determined as the foreign object detection result of the area to be tested; or, the two-dimensional detection result is re-examined to obtain a second verification test result, and the second verification test result is determined as the foreign object detection result of the area to be tested.

[0017] In one embodiment, the processing unit re-examines the three-dimensional detection result in the following manner to obtain a first verification detection result: converting the foreign object data points in the two-dimensional detection result into three-dimensional foreign object data points; determining the three-dimensional re-examined foreign object data points, wherein the three-dimensional re-examined foreign object data points are foreign object data points in the three-dimensional foreign object data points that are different from the foreign object data points in the three-dimensional detection result; re-examining the three-dimensional re-examined foreign object data points to obtain a three-dimensional re-examine result; and merging the three-dimensional re-examine result and the three-dimensional detection result to obtain the first verification detection result.

[0018] In one embodiment, the processing unit is further configured to: adjust the detection parameters used for three-dimensional detection based on the first verification detection result.

[0019] In one embodiment, the two-dimensional detection result includes a two-dimensional AI detection result and a two-dimensional edge detection result. The processing unit re-examines the two-dimensional detection result in the following manner to obtain a second verification detection result: converting foreign object data points in the three-dimensional detection result into two-dimensional foreign object data points; determining two-dimensional re-examined foreign object data points, wherein the two-dimensional re-examined foreign object data points are foreign object data points in the two-dimensional foreign object data points that are different from those in the two-dimensional AI detection result, and foreign object data points in the two-dimensional edge detection result that are different from those in the two-dimensional AI detection result; re-examining the two-dimensional re-examined data points based on an unsupervised detection network to determine the two-dimensional AI re-examined result; and merging the two-dimensional AI re-examined result and the two-dimensional AI detection result to obtain the second verification detection result.

[0020] In one embodiment, the processing unit is further configured to: update the first detection parameter to a second detection parameter in response to the presence of foreign object data points in the three-dimensional re-inspection result; the first detection parameter is a three-dimensional detection parameter used when detecting foreign objects in the area to be tested based on the three-dimensional point cloud data, and the second detection parameter is a detection parameter used when re-inspecting the three-dimensional re-inspection foreign object data points; the parameter value corresponding to the first detection parameter is greater than the parameter value corresponding to the second detection parameter.

[0021] According to a third aspect of the present disclosure, a foreign object detection system for a test area is provided, which uses the foreign object detection method for a test area as described in the first aspect or any one of the first aspects to detect the test area.

[0022] According to a fourth aspect of the present disclosure, a foreign object detection device for a test area is provided, comprising: a processor: a memory for storing processor-executable instructions; wherein the processor is configured to: execute the foreign object detection method for a test area as described in the first aspect or any one of the first aspects.

[0023] According to a fifth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the foreign object detection method for the test area as described in the first aspect or any one of the first aspects.

[0024] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: 3D foreign object detection based on 3D point cloud data of the area to be tested acquired by an image device. Converting the 3D point cloud data into 2D image data for 2D foreign object detection eliminates the need to separately acquire 2D image data of the area to be tested, thus reducing detection costs and improving foreign object detection efficiency.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] Figure 1 This is a block diagram illustrating a target detection based on visual analysis according to an exemplary embodiment of the present disclosure.

[0028] Figure 2 This is a flowchart illustrating a method for detecting foreign objects in a test area according to an exemplary embodiment.

[0029] Figure 3A block diagram illustrating a method for detecting foreign objects in a test area according to an exemplary embodiment of the present disclosure.

[0030] Figure 4 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to an exemplary embodiment.

[0031] Figure 5 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to an exemplary embodiment.

[0032] Figure 6 This is a flowchart illustrating a method for merging foreign object data points according to an exemplary embodiment.

[0033] Figure 7 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to yet another exemplary embodiment.

[0034] Figure 8 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to yet another exemplary embodiment.

[0035] Figure 9 A block diagram illustrating a method for determining the foreign object detection result of a region to be tested, according to an exemplary embodiment of the present disclosure.

[0036] Figure 10 A block diagram illustrating an optimized three-dimensional detection method according to an exemplary embodiment of the present disclosure.

[0037] Figure 11 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to yet another exemplary embodiment.

[0038] Figure 12 A block diagram illustrating a method for determining the foreign object detection result of a region to be tested, according to an exemplary embodiment of the present disclosure.

[0039] Figure 13 This is a flowchart illustrating a method for optimizing a detection model according to yet another exemplary embodiment.

[0040] Figure 14 This is a block diagram illustrating an optimization and iterative method for a two-dimensional AI detection model according to an exemplary embodiment.

[0041] Figure 15 This is a block diagram of a foreign object detection device for a test area according to an exemplary embodiment.

[0042] Figure 16 This is a block diagram illustrating an apparatus for detecting foreign objects in a test area according to an exemplary embodiment. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0044] The foreign object detection method for the area to be tested provided in this disclosure is applied to scenarios where foreign objects are present in the area.

[0045] In an exemplary embodiment of this disclosure, foreign object detection is performed on terminal devices such as tablets and mobile phones before assembly. If foreign objects such as screws, terminals, metal shavings, release paper, or hair remain inside the terminal device during assembly, assembly may fail, or components may break after assembly, reducing the terminal's lifespan and potentially causing injury to the user due to component damage (e.g., battery) during use. Therefore, it is necessary to detect foreign objects on the components of the device to be assembled before assembly.

[0046] In an exemplary embodiment of this disclosure, foreign object detection can be performed on the battery compartment before the terminal battery is assembled.

[0047] Most related technologies detect foreign objects in the test area through hardware mechanisms such as displacement sensors, infrared, or ultrasound, or through manual visual inspection by personnel. However, the detection accuracy of manual methods and the aforementioned hardware mechanisms is limited, they cannot detect tiny objects, and the detection efficiency is low. In summary, methods for detecting foreign objects through manual visual inspection or hardware mechanisms suffer from low detection accuracy, high false negative rates, high manpower consumption, and low productivity.

[0048] In view of this, the present disclosure provides a method for detecting foreign objects in a test area. In this method, visual analysis is used to detect foreign objects present in the test area. Detecting foreign objects in a test area through visual analysis can detect foreign objects based on the acquired image data of the test area, enabling the detection of minute foreign objects, improving detection accuracy, reducing the false negative rate, and saving manpower.

[0049] In one implementation, a 2D vision detection method based on a 2D camera and a 3D vision detection method based on a 3D camera are used for joint visual analysis.

[0050] like Figure 1The block diagram for object detection based on visual analysis is shown. In related object detection technologies, a 3D detection system performs 3D object detection, generating and outputting 3D detection results. A 2D detection system performs 2D object detection, generating and outputting 2D detection results. Based on the 2D and 3D detection results, the final foreign object detection result is determined.

[0051] like Figure 1 The block diagram of target detection based on visual analysis is shown in the figure. The detection method combining 2D and 3D in the relevant target detection technology is as follows: (1) Image acquisition: 2D image of the target is acquired through a 2D camera and 3D image of the target is acquired through a 3D camera. Based on the acquired 2D and 3D images of the target, 2D feature data and 3D feature data of the target are acquired. (2) 2D target detection and 3D point cloud data processing: Using a detection network model trained based on deep learning, the acquired 2D target image is subjected to various two-dimensional geometric analyses such as missing / presence detection, discrete object analysis, pattern alignment, barcode and optical character recognition, and edge detection to generate 2D analysis results. The 3D point cloud data is preprocessed to eliminate interference data. Based on the reference plane of the reference target where there are no foreign objects, the acquired 3D point cloud data is fitted to a plane to determine the distance between each data in the 3D point cloud data and the reference plane. (3) Determine the presence of the target based on 2D analysis results and 3D point cloud data: Determine the presence of foreign objects based on the analysis results of the detection network model, and determine the number and location of foreign objects if they are detected. Also, identify foreign object data points in the 3D point cloud data whose distance from the reference plane is greater than a distance threshold. The distance threshold is the critical distance value for determining whether a data point is a target data point; point cloud data points with a distance greater than the threshold are foreign object data points. (4) Output detection results: Output the analysis results of the detection network model as 2D detection results, and output the foreign object data points in the 3D point cloud data whose distance from the reference plane is greater than the distance threshold as 3D detection results. (5) Summarize detection results: Convert the 2D and 3D detection results into the same data format (2D data or 3D data), take the union, and output it as the foreign object detection result.

[0052] Detection methods combining 2D and 3D vision can complement each other. 3D detection methods can detect foreign objects that 2D methods cannot (such as foreign objects not yet learned by the 2D detection model, or foreign objects similar to the background), while 2D detection methods can detect foreign objects that 3D methods cannot (such as foreign objects below a certain height on an uneven surface). In summary, using a combined 3D and 2D vision detection method can reduce the false negative rate. However, using both 2D and 3D cameras simultaneously for foreign object detection increases the space required for camera movement mechanisms and the time required for data acquisition and processing. Therefore, 2D / 3D combined vision analysis methods based on 2D and 3D cameras suffer from high cost and low productivity.

[0053] In view of this, this disclosure proposes a method for foreign object detection in a test area, which performs three-dimensional detection and two-dimensional detection on the data of the test area acquired by the same image acquisition device to obtain the foreign object detection results of the test area, thereby reducing detection costs and improving detection efficiency.

[0054] In one approach, a 2D image of the area to be tested is acquired using a 2D camera, 2D image data is extracted from the 2D image, and the 2D image data is converted into 3D point cloud data for 2D and 3D combined visual inspection.

[0055] Another approach involves acquiring a 3D image of the area to be tested using a 3D camera, extracting 3D point cloud data from the 3D image, converting the 3D point cloud data into 2D image data, and then performing visual inspection combining 2D and 3D data.

[0056] It is understandable that 3D images contain more information about the area to be tested than 2D images. Acquiring 3D point cloud data and 2D image data for foreign object detection based on 3D images can reduce the false negative rate. Therefore, this disclosure uses a 3D image acquisition device to acquire 3D images of the area to be tested and 3D point cloud data of the area to be tested, and then performs 3D and 2D combined foreign object detection.

[0057] Figure 2 This is a flowchart illustrating a method for detecting foreign objects in a test area according to an exemplary embodiment. Figure 2 As shown, the method includes steps S101 to S104.

[0058] In step S101, three-dimensional point cloud data of the area to be measured is acquired.

[0059] In this embodiment of the disclosure, a three-dimensional image acquisition device is used to acquire a three-dimensional image of the area to be detected, and three-dimensional point cloud data of the area to be detected is acquired based on the three-dimensional image of the area to be detected.

[0060] Among them, the three-dimensional point cloud data of the area to be tested is a coordinate set that fits the surface of the area to be tested, and each data point in the three-dimensional point cloud data is distributed in a regular manner.

[0061] In step S102, based on the three-dimensional point cloud data, the two-dimensional image data of the area to be measured is determined.

[0062] In this embodiment, the two-dimensional image data is determined based on the acquired three-dimensional point cloud data and the correspondence between the three-dimensional point cloud data and the two-dimensional image data. Each data point in the two-dimensional image data has a corresponding data point in the three-dimensional point cloud data. This conversion of the three-dimensional point cloud data avoids the need for additional two-dimensional image acquisition devices to obtain the two-dimensional image data.

[0063] In step S103, foreign object detection is performed on the area to be tested based on three-dimensional point cloud data to obtain three-dimensional detection results, and foreign object detection is performed on the area to be tested based on two-dimensional image data to obtain two-dimensional detection results.

[0064] In step S104, the foreign object detection result of the area to be tested is determined based on the three-dimensional detection result and the two-dimensional detection result.

[0065] In this embodiment of the disclosure, three-dimensional detection and two-dimensional detection are performed based on the three-dimensional point cloud data of the area to be tested collected by the image device, and the three-dimensional detection results and two-dimensional detection results are obtained, thereby obtaining the foreign object detection results of the area to be tested, which can reduce detection costs and improve detection efficiency.

[0066] The following embodiments of this disclosure illustrate the process for determining the foreign object detection result of the test area. The process for determining the foreign object detection result of the test area includes: acquiring three-dimensional point cloud data; converting the three-dimensional point cloud data into two-dimensional image data; acquiring three-dimensional detection results based on the three-dimensional point cloud data; acquiring two-dimensional detection results based on the two-dimensional image data; and performing joint analysis to output the foreign object detection result.

[0067] In this embodiment, there is a correspondence between the data points in the 3D point cloud data and the 2D image data. The 3D point cloud data can be converted into 2D image data based on a preset conversion model. All spatial coordinate data in the 3D point cloud data is converted into a grayscale image containing image depth information; different grayscale values ​​correspond to different image depths. A projection plane is determined in the coordinate system of the point cloud data. The corresponding position and distance of each data point in the point cloud data on the projection plane are determined. The distance between the data point and the projection plane is the image depth. Different grayscale values ​​are used to represent the distance between the data point and the projection plane. The image data containing image depth information corresponding to the point cloud data on the projection plane is determined as the 2D image data of the area to be measured.

[0068] It is understood that, in order to facilitate two-dimensional foreign object detection, the projection surface used to acquire two-dimensional image data in this disclosure can be a plane that is horizontal to the plane where the area to be measured is located, and the acquired two-dimensional image data is approximately a top view of the area to be measured.

[0069] In an exemplary embodiment of this disclosure, such as Figure 3 The flowchart of the foreign object detection method for the area to be tested is shown. After starting the foreign object detection in the area to be tested, the parameters for 3D and 2D foreign object detection are manually configured, and the 3D point cloud data is collected. The process for 3D foreign object detection is as follows:

[0070] (1) Point cloud data preprocessing: The point cloud data is filtered to remove scattered points and isolated points that serve as interference data, thereby reducing interference in the subsequent 3D foreign object detection process. Scattered points and isolated points are image noise in the 3D image and are irregularly distributed in the point cloud data, while the arrangement and spacing of normally acquired point cloud data are regularly distributed. The filter can remove scattered points and isolated points that serve as interference data in the 3D point cloud data.

[0071] (2) Plane Fitting: A standard reference plane is selected according to preset parameters and plane fitting is performed. The preset parameters are the parameters of the test area without foreign objects, and the reference plane is the plane parameters of the test area without foreign objects. This disclosure performs plane fitting based on the acquired 3D point cloud data of the test area and the reference data of the test area without foreign objects, so that the acquired 3D point cloud data is distributed around the reference plane.

[0072] (3) Threshold Judgment: The area to be measured is searched and calculated, and compared with a set threshold. The set threshold is the critical distance value for judging whether a data point is a foreign object data point. Data points whose distance from the reference plane is greater than the threshold are foreign object data points. Foreign object data points are determined by judging the distance between each data point in the acquired 3D point cloud data and the reference plane.

[0073] (4) Three-dimensional detection results: Statistical analysis of the detection results based on the threshold judgment, marking the areas where data points exceeding the threshold are located and outputting the location information of foreign objects, and counting the number and location of foreign object data points.

[0074] In an exemplary embodiment of this disclosure, such as Figure 3 The block diagram of the foreign object detection method in the test area is shown. After generating two-dimensional image data based on three-dimensional point cloud data, the two-dimensional foreign object detection includes two-dimensional edge detection and two-dimensional AI detection. Before the two detections, the two-dimensional image data needs to be preprocessed, including image filtering to eliminate the interference of isolated points in the image data; and contrast enhancement to improve the recognition of image features.

[0075] In an exemplary embodiment of this disclosure, such as Figure 3 The flowchart of the foreign object detection method in the test area is shown. The process of performing two-dimensional AI detection is as follows:

[0076] (1) AI Object Detection: Using deep learning-based object detection algorithms to detect various features of two-dimensional image data, performing missing / presence detection, discrete object analysis, pattern alignment, barcode and optical character recognition, and various two-dimensional geometric analyses such as edge detection. Detecting pre-labeled targets in two-dimensional image data.

[0077] (2) Two-dimensional AI detection results: Output abnormal data points among multiple detection target points.

[0078] Understandably, object detection algorithms require pre-labeling of sample data to learn the possible locations and types of foreign objects. Therefore, 2D AI detection methods can only detect labeled targets.

[0079] In an exemplary embodiment of this disclosure, such as Figure 3 The flowchart of the foreign object detection method in the test area is shown. The process of two-dimensional edge detection is as follows:

[0080] (1) Edge detection: Based on the parameters pre-configured by the user, the grayscale image is segmented to obtain multiple segmented images. The edge features of each segmented image are filtered to determine the edge length.

[0081] (2) Edge filtering: Regions with an edge length greater than the threshold are identified as foreign object edges.

[0082] (3) Two-dimensional edge detection results: Record the position and size information corresponding to each foreign object edge.

[0083] In this embodiment, since the lasers of the 3D image acquisition device are all blue or red, they are transparent when encountering blue or red objects, resulting in omissions in the acquired 3D point cloud data and the converted 2D brightness map, thus leading to false negatives. This disclosure performs 2D AI detection while also performing 2D edge detection based on the same 2D image data, further reducing the false negative rate.

[0084] In this embodiment, the foreign object data points in the two-dimensional and three-dimensional detection results differ, and their data representation formats also differ. Based on these differences in content and presentation, the final foreign object detection results for the target area can be output in multiple ways. Furthermore, different output methods can be selected based on user needs. If the requirement is simply to obtain the detection results, only the union of the three-dimensional and two-dimensional detection results needs to be output. If the requirement is to optimize both two-dimensional and three-dimensional detection, then mutual verification between the three-dimensional and two-dimensional detection results is necessary, followed by optimization of both based on the verification results.

[0085] Figure 4 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to an exemplary embodiment. Figure 4 As shown, the method includes steps S201 to S202.

[0086] In step S201, in response to obtaining two-dimensional detection results and three-dimensional detection results.

[0087] In step S202A, the foreign object data points in the three-dimensional detection results and the foreign object data points in the two-dimensional detection results are merged, and the merged foreign object data points are determined as the foreign object detection results of the area to be tested.

[0088] In this embodiment of the disclosure, if the purpose of detection is only to obtain foreign object detection results, the union of foreign object data points in the three-dimensional detection results and foreign object data points in the two-dimensional detection results can be directly output as the foreign object detection results of the area to be tested.

[0089] In step S202B, the three-dimensional detection result or the two-dimensional detection result is reviewed and the review result is determined as the foreign object detection result of the area to be tested.

[0090] S202A and S202B are optional execution methods.

[0091] In this embodiment, both the 3D and 2D detection results have missed detections, and there is a complementary relationship between them. The 2D and 3D detection results can be verified separately based on this complementary relationship, and the 2D and 3D detection methods can be optimized based on the verification results.

[0092] In this embodiment of the disclosure, such as Figure 5 The flowchart of the method for determining the foreign object detection result of the test area is shown. This disclosure can directly take the union of the three-dimensional detection results and the two-dimensional detection results as the foreign object detection result of the test area. When there is no need to optimize the two-dimensional and three-dimensional detection methods, and only the foreign object information of the test area needs to be obtained, the foreign object detection result of the test area can be quickly determined by merging the detection results.

[0093] In this embodiment, the three-dimensional detection results are expressed as point cloud data, and the two-dimensional detection results are expressed as two-dimensional image data. Due to the error in the expression format between the three-dimensional and two-dimensional detection results, this disclosure requires that the three-dimensional and two-dimensional detection results be converted into the same expression format (such as two-dimensional image data or three-dimensional point cloud data) before they can be merged. The following embodiments of this disclosure illustrate the method for merging foreign object data points.

[0094] Figure 6 This is a flowchart illustrating a method for merging foreign object data points according to an exemplary embodiment. Figure 6 As shown, the method includes steps S301, S302A, and S302B.

[0095] In step S301, in response to obtaining two-dimensional detection results and three-dimensional detection results.

[0096] In step S302A, the foreign object data points in the three-dimensional detection results are converted into two-dimensional foreign object data points, and the union of the two-dimensional foreign object data points and the foreign object data points in the two-dimensional detection results is taken as the merged foreign object data points.

[0097] In step S302B, the foreign object data points in the two-dimensional detection results are converted into three-dimensional foreign object data points, and the union of the three-dimensional foreign object data points and the foreign object data points in the three-dimensional detection results is taken as the merged foreign object data points.

[0098] Among them, steps S302A and S302B are optional execution methods.

[0099] In this embodiment of the disclosure, the foreign object data points contained in the two-dimensional detection results and the three-dimensional detection results are converted into point cloud data or two-dimensional image data, and then merged. The union of the foreign object data points obtained after merging is the foreign object detection result of the area to be tested.

[0100] In this embodiment, the two-dimensional and three-dimensional detection results can be cross-verified based on their differences, thereby improving the accuracy of the final output foreign object detection result. When determining the foreign object detection result for the area to be tested, this embodiment can cross-verify the two-dimensional and three-dimensional detection results to obtain foreign object data points not present in the original detection results, enriching the original two-dimensional and three-dimensional detection results and reducing the false negative rate. Furthermore, the relevant parameters of the two-dimensional and three-dimensional detection can be adjusted based on the verified detection results, thereby optimizing the two-dimensional and three-dimensional detection methods.

[0101] In this embodiment of the disclosure, the two-dimensional detection results can be verified based on the three-dimensional detection results, thereby improving the accuracy of the two-dimensional detection results. The following embodiments of this disclosure further illustrate the process for determining the foreign object detection results of the area to be tested.

[0102] Figure 7 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to yet another exemplary embodiment. For example... Figure 7 As shown, the method includes steps S401, S402A, and S402B.

[0103] In step S401, in response to obtaining two-dimensional detection results and three-dimensional detection results.

[0104] In step S402A, the three-dimensional detection results are re-examined to obtain the first verification detection result, and the first verification detection result is determined as the foreign object detection result of the area to be tested.

[0105] In this embodiment of the disclosure, a secondary verification is performed on point cloud data other than foreign object data points in the 3D detection results to further reduce the false negative rate of the 3D detection results.

[0106] In step S402B, the two-dimensional detection result is re-examined to obtain a second verification detection result, which is then determined as the foreign object detection result for the area to be tested.

[0107] Among them, steps S402A and S402B are optional execution methods.

[0108] In this embodiment of the disclosure, by performing secondary verification on the two-dimensional and three-dimensional detection results, other data points besides the foreign object data points are detected, thereby improving the accuracy of the two-dimensional and three-dimensional detection results respectively.

[0109] In this embodiment of the disclosure, the three-dimensional detection results can be verified based on the two-dimensional detection results, thereby improving the accuracy of the three-dimensional detection results. The following embodiments of this disclosure further illustrate the method for determining the foreign object detection results of the area to be tested.

[0110] Figure 8 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to yet another exemplary embodiment. For example... Figure 8 As shown, the method includes steps S501 to S504.

[0111] In step S501, the foreign object data points in the two-dimensional detection results are converted into three-dimensional foreign object data points.

[0112] In this embodiment of the disclosure, the foreign object data points in the two-dimensional detection results converted into three-dimensional foreign object data can be intuitively compared with the foreign object data points in the three-dimensional detection results.

[0113] In step S502, the data points for the three-dimensional re-inspection of foreign objects are determined.

[0114] Among them, the three-dimensional re-inspection foreign object data points are foreign object data points that exist in the two-dimensional detection results but not in the three-dimensional detection results, and are three-dimensional point cloud data points that may contain foreign objects.

[0115] In step S503, the three-dimensional re-inspection foreign object data points are re-inspected to obtain the three-dimensional re-inspection results.

[0116] In this embodiment of the disclosure, after determining the three-dimensional re-inspection foreign object data points, the detection parameters are adjusted for the three-dimensional re-inspection foreign object data points, and three-dimensional foreign object detection is performed again. After performing plane fitting, based on the user-configured re-inspection threshold parameters, a smaller filter is used, and based on a smaller distance threshold, data points in each of the three-dimensional re-inspection foreign object data points that are greater than the distance threshold are identified, and the set of data points in the three-dimensional re-inspection foreign object data points that are greater than the distance threshold is determined as the three-dimensional re-inspection result.

[0117] In step S504, the three-dimensional re-inspection results and the three-dimensional detection results are merged to obtain the first verification detection result.

[0118] In this embodiment of the disclosure, the three-dimensional detection results are examined based on the two-dimensional detection results to determine the three-dimensional re-inspection data points that may contain foreign objects, other than the foreign object data points in the three-dimensional detection results. The three-dimensional re-inspection data is then re-inspected based on a smaller threshold to obtain the three-dimensional re-inspection results. The union of the three-dimensional re-inspection results and the original three-dimensional detection results is determined as the three-dimensional verification detection results, thereby improving the accuracy of three-dimensional foreign object detection.

[0119] Understandably, the threshold used for 3D detection can be adaptively adjusted based on the results of 3D re-inspection, thereby making subsequent 3D detection more accurate and reducing the false negative rate of 3D detection.

[0120] In an exemplary embodiment of this disclosure, such as Figure 9The flowchart illustrates a method for determining foreign object detection results in a test area. The method involves verifying the 3D detection results based on the 2D detection results and outputting the foreign object detection results for the test area as follows: Foreign object location information (i.e., foreign object data points) that exist in the 2D detection results but do not belong to the 3D detection results are identified. The target foreign object region is extracted from the 3D point cloud data, and 3D re-inspection data points that differ from the 3D detection results are determined. A smaller filter size and a smaller re-inspection distance threshold are used for filtering, point cloud segmentation is performed, and based on the distance between the 3D re-inspection data points and the reference plane, combined with the re-inspection distance threshold, it is determined whether new foreign object data points exist. Data points whose distance from the reference plane is greater than the re-inspection distance threshold are identified as re-inspection data points, thus determining the 3D re-inspection results. The re-inspection data points detected by the 3D re-inspection results are merged with the foreign object data points in the original 3D detection results, and the union of the 3D re-inspection results and the original 3D detection results is the foreign object detection result for the test area.

[0121] In this embodiment of the disclosure, based on the three-dimensional re-inspection results obtained from each re-inspection of the three-dimensional re-inspection foreign object data points, the three-dimensional detection parameters can be adjusted to optimize the three-dimensional detection effect and reduce the false negative rate of three-dimensional detection. The following embodiments of this disclosure illustrate the method for optimizing three-dimensional detection.

[0122] Figure 10 This is a flowchart illustrating an optimized 3D detection method according to an exemplary embodiment. Figure 10 As shown, the method includes steps S601 to S602.

[0123] In step S601, the first verification test result is obtained.

[0124] In step S602, in response to the presence of foreign object data points in the three-dimensional re-inspection results, the first detection parameter is updated to the second detection parameter.

[0125] The first detection parameter is a three-dimensional detection parameter used when detecting foreign objects in the area to be tested based on three-dimensional point cloud data, and the second detection parameter is a detection parameter used when re-inspecting the three-dimensional re-inspection foreign object data points; the parameter value corresponding to the first detection parameter is greater than the parameter value corresponding to the second detection parameter.

[0126] In this embodiment, both the first and second detection parameters are distance thresholds used to determine whether a 3D data point is a foreign object data point after planar fitting during 3D detection. When performing 3D detection on foreign object data points for re-inspection, after planar fitting, a distance threshold smaller than that used in the original 3D detection process must be used for re-inspection. After obtaining the 3D re-inspection results, the results can be manually verified. If the verification shows that a corresponding foreign object exists at the corresponding position of the detected foreign object data point in the area to be tested, the re-inspection results are accurate. Based on the detection threshold used during re-inspection, the detection threshold for joint detection can be adaptively adjusted, i.e., the filter size used to filter foreign object data points can be modified to optimize the 3D detection method and reduce the false negative rate of 3D detection.

[0127] In this embodiment of the disclosure, the two-dimensional detection results can be verified based on the three-dimensional detection results, thereby improving the accuracy of the two-dimensional detection results. The following embodiments of this disclosure further illustrate the method for determining the foreign object detection results of the area to be tested.

[0128] Figure 11 This is a flowchart illustrating a method for determining the foreign object detection result of a test area according to yet another exemplary embodiment. For example... Figure 11 As shown, the method includes steps S701 to S704.

[0129] In step S701, the foreign object data points in the three-dimensional detection results are converted into two-dimensional foreign object data points.

[0130] In this embodiment of the disclosure, the foreign object data points in the three-dimensional detection results converted into two-dimensional foreign object data can be intuitively compared with the foreign object data points in the two-dimensional detection results.

[0131] In step S702, the two-dimensional re-inspection foreign object data points are determined.

[0132] Among them, the two-dimensional re-inspection foreign object data points are other foreign object data points in the two-dimensional foreign object data points that are different from the foreign object data points in the two-dimensional AI detection results, and other foreign object data points in the two-dimensional edge detection results that are different from the foreign object data points in the two-dimensional AI detection results.

[0133] In this embodiment of the disclosure, the data points for two-dimensional re-inspection of foreign objects exist in the three-dimensional detection results and the two-dimensional edge detection results, but not in the two-dimensional AI detection results. These are two-dimensional image data points that may contain foreign objects.

[0134] In step S703, the two-dimensional re-inspection data points are re-inspected based on the unsupervised detection network to determine the two-dimensional AI re-inspection result.

[0135] In this embodiment of the disclosure, after determining the two-dimensional re-inspection foreign object data points, two-dimensional foreign object detection is performed again on the two-dimensional re-inspection foreign object data points. After preprocessing such as image filtering and contrast enhancement, the two-dimensional re-inspection foreign object data is re-inspected based on an unsupervised network trained only on data from the region without foreign objects to be tested. Data points that are abnormal or differ from the data from the region without foreign objects to be tested are identified, and the abnormal data points are determined as the two-dimensional re-inspection results.

[0136] In step S704, the two-dimensional AI re-inspection result and the two-dimensional AI detection result are merged to obtain the second re-inspection result.

[0137] In this embodiment of the disclosure, the two-dimensional detection results are examined based on the three-dimensional detection results. Secondary re-inspection data points that may contain foreign objects are identified in addition to the foreign object data points in the two-dimensional detection results. The two-dimensional re-inspection data are then re-inspected based on an unsupervised network to obtain the two-dimensional re-inspection results. The union of the two re-inspection results and the original two-dimensional detection results is determined as the two-dimensional verification detection result, thereby improving the accuracy of two-dimensional foreign object detection.

[0138] In an exemplary embodiment of this disclosure, such as Figure 12 The flowchart of the method for determining the foreign object detection result of the test area is shown. The method involves verifying the two-dimensional detection result based on the three-dimensional detection result and outputting the foreign object detection result of the test area as follows: Locate the location information of foreign objects present in the three-dimensional detection result and the two-dimensional edge detection result, but not in the two-dimensional AI detection result; extract the target area corresponding to the two-dimensional re-inspection data points in the two-dimensional image data, perform image filtering, contrast enhancement, and other processing sequentially, and then use an unsupervised deep learning-based network (trained only on test area samples without foreign objects) to determine the presence or absence of foreign objects; merge the two-dimensional re-inspection result with the original two-dimensional detection result and output the result; the union of the two-dimensional re-inspection result and the original two-dimensional detection result is the foreign object detection result of the test area.

[0139] In this embodiment of the disclosure, the detection model for two-dimensional foreign object detection can be iteratively optimized based on the two-dimensional re-inspection data points obtained during each foreign object detection process in the area to be tested.

[0140] Figure 13 This is a flowchart illustrating a method for optimizing a detection model according to yet another exemplary embodiment. For example... Figure 13 As shown, the method includes steps S801 to S802.

[0141] In step S801, the second verification test result is obtained.

[0142] In this embodiment, the two-dimensional AI detection algorithm can only detect pre-labeled data points and cannot detect other data points besides the pre-labeled data points. The second verification detection result includes foreign object data points other than the pre-labeled data points.

[0143] In step S802, foreign object detection points for two-dimensional AI detection are determined from the two-dimensional re-inspection results, and the foreign object detection points are added to the two-dimensional AI training samples.

[0144] In this embodiment of the disclosure, such as Figure 14 The flowchart of the optimization and iteration method of the 2D AI detection model is shown. When reviewing the 2D AI detection results, for foreign object data points that exist in the 3D detection results or 2D edge detection results but not in the 2D AI detection results, they are mapped to the 2D image and recorded as annotation information. The manual review determines whether to add them to the target detection network training dataset, thus achieving the effect of automatic sample data amplification and automatic annotation. Based on the amplified and annotated dataset and the original data samples, the model is iteratively optimized again. After optimization, the model is deployed to improve the accuracy of the 2D AI detection results and reduce the false negative rate of 2D AI detection.

[0145] In this embodiment of the present disclosure, three-dimensional point cloud data of the area to be tested is acquired by a three-dimensional imaging device, and two-dimensional image data of the area to be tested is determined. Foreign object detection is performed on the area to be tested based on the three-dimensional point cloud data and the two-dimensional image data, respectively, to obtain three-dimensional detection results and two-dimensional detection results. The foreign object detection results are obtained based on the three-dimensional detection results and the two-dimensional detection results in the following manner: (1) Foreign object data points in the three-dimensional detection results and the two-dimensional detection results are converted into three-dimensional point cloud data or two-dimensional image data and then the union is taken as the foreign object detection result of the area to be tested. (2) Three-dimensional re-inspection is performed based on foreign object data points in the two-dimensional detection results that are different from the three-dimensional detection results, and the union of the three-dimensional re-inspection results and the original three-dimensional detection results is taken as the foreign object detection result of the area to be tested. (3) Two-dimensional re-inspection is performed based on the three-dimensional detection results and the two-dimensional edge detection results that are different from the two-dimensional detection results, and the union of the two-dimensional re-inspection results and the original two-dimensional detection results is taken as the foreign object detection result of the area to be tested. New foreign object data points for two-dimensional AI detection are determined in the two-dimensional re-inspection results and added to the two-dimensional AI detection model. This disclosure allows for the performance of 3D and 2D detection based on 3D point cloud data of the area to be tested acquired by an image device, thereby obtaining foreign object detection results for the area to be tested. This reduces the rate of missed detections, lowers detection costs, and improves detection efficiency.

[0146] Based on the same concept, this disclosure also provides a foreign object detection device 100 for a test area.

[0147] It is understood that the foreign object detection device 100 for the test area provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0148] Figure 15 This is a block diagram illustrating a foreign object detection device 100 for a test area according to an exemplary embodiment. (Refer to...) Figure 15 The device includes a data acquisition unit 101, a conversion unit 102, a determination unit 103, and a processing unit 104.

[0149] The acquisition unit 101 is used to acquire three-dimensional point cloud data of the area to be measured.

[0150] The conversion unit 102 is used to determine the two-dimensional image data of the area to be measured based on the three-dimensional point cloud data.

[0151] The determination unit 103 is used to perform foreign object detection on the area to be tested based on three-dimensional point cloud data to obtain three-dimensional detection results, and to perform foreign object detection on the area to be tested based on two-dimensional image data to obtain two-dimensional detection results.

[0152] The processing unit 104 is used to determine the foreign object detection result of the area to be tested based on the three-dimensional detection result and the two-dimensional detection result.

[0153] In one embodiment, the processing unit 104 determines the foreign object detection result of the area to be tested based on the three-dimensional detection result and the two-dimensional detection result in the following manner: merging the foreign object data points in the three-dimensional detection result and the foreign object data points in the two-dimensional detection result, and determining the merged foreign object data points as the foreign object detection result of the area to be tested; or performing a verification detection on the three-dimensional detection result or the two-dimensional detection result, and determining the verification detection result as the foreign object detection result of the area to be tested.

[0154] In one embodiment, the processing unit 104 merges the foreign object data points in the three-dimensional detection results and the foreign object data points in the two-dimensional detection results in the following manner: converting the foreign object data points in the three-dimensional detection results into two-dimensional foreign object data points, and taking the union of the two-dimensional foreign object data points and the foreign object data points in the two-dimensional detection results as the merged foreign object data points; or converting the foreign object data points in the two-dimensional detection results into three-dimensional foreign object data points, and taking the union of the three-dimensional foreign object data points and the foreign object data points in the three-dimensional detection results as the merged foreign object data points.

[0155] In one embodiment, the processing unit 104 performs a verification test on the three-dimensional detection result or the two-dimensional detection result in the following manner: the three-dimensional detection result is re-examined to obtain a first verification test result, and the first verification test result is determined as the foreign object detection result of the area to be tested; or, the two-dimensional detection result is re-examined to obtain a second verification test result, and the second verification test result is determined as the foreign object detection result of the area to be tested.

[0156] In one embodiment, the processing unit 104 performs a re-inspection on the three-dimensional detection result in the following manner to obtain a first re-inspection result: converting the foreign object data points in the two-dimensional detection result into three-dimensional foreign object data points; determining the three-dimensional re-inspection foreign object data points, wherein the three-dimensional re-inspection foreign object data points are foreign object data points in the three-dimensional foreign object data points that are different from the foreign object data points in the three-dimensional detection result; re-inspecting the three-dimensional re-inspection foreign object data points to obtain a three-dimensional re-inspection result; merging the three-dimensional re-inspection result and the three-dimensional detection result to obtain the first re-inspection result.

[0157] In one embodiment, the processing unit 104 is further configured to: update the first detection parameter to the second detection parameter in response to the presence of foreign object data points in the three-dimensional re-inspection result; the first detection parameter is the three-dimensional detection parameter used when performing foreign object detection on the area to be tested based on three-dimensional point cloud data, and the second detection parameter is the detection parameter used when re-inspecting the three-dimensional re-inspection foreign object data points; the parameter value corresponding to the first detection parameter is greater than the parameter value corresponding to the second detection parameter.

[0158] In one embodiment, the two-dimensional detection result includes a two-dimensional AI detection result and a two-dimensional edge detection result. The processing unit 104 re-examines the two-dimensional detection result in the following manner to obtain a second verification detection result: the foreign object data points in the three-dimensional detection result are converted into two-dimensional foreign object data points; the two-dimensional re-examined foreign object data points are determined, which are foreign object data points in the two-dimensional foreign object data points that are different from those in the two-dimensional AI detection result, and foreign object data points in the two-dimensional edge detection result that are different from those in the two-dimensional AI detection result; the two-dimensional re-examined data points are re-examined based on an unsupervised detection network to determine the two-dimensional AI re-examined result; the two-dimensional AI re-examined result and the two-dimensional AI detection result are merged to obtain the second verification detection result.

[0159] In one embodiment, the processing unit 104 is further configured to: determine foreign object detection points for two-dimensional AI detection in the two-dimensional re-inspection results, and add the foreign object detection points to the two-dimensional AI training samples, wherein the two-dimensional AI training samples are used to train the two-dimensional AI detection model, and the two-dimensional AI detection model is used to determine the two-dimensional AI detection results.

[0160] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0161] Figure 16 This is a block diagram illustrating an apparatus 200 for detecting foreign objects in a test area according to an exemplary embodiment. The apparatus 200 can be provided as a terminal. For example, the apparatus 200 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0162] Reference Figure 16 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0163] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0164] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0165] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0166] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0167] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0168] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0169] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0170] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0171] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0172] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0173] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0174] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0175] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0176] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0177] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0178] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0179] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for detecting foreign objects in a test area, characterized in that, include: A three-dimensional image acquisition device is used to acquire a three-dimensional image of the area to be tested, and three-dimensional point cloud data of the area to be tested is acquired based on the three-dimensional image. Based on the correspondence between 3D point cloud data and 2D image data, the 2D image data corresponding to the 3D point cloud data is determined, wherein the 3D point cloud data includes the data points corresponding to each data point in the 2D image data; Foreign object detection is performed on the area to be tested based on the three-dimensional point cloud data to obtain a three-dimensional detection result, and foreign object detection is performed on the area to be tested based on the two-dimensional image data to obtain a two-dimensional detection result. The two-dimensional detection result includes two-dimensional AI detection result and two-dimensional edge detection result. The three-dimensional detection result or the two-dimensional detection result is verified, and the verified detection result is determined as the foreign object detection result of the area to be tested. The step of verifying the three-dimensional detection result or the two-dimensional detection result includes: The three-dimensional detection results are re-examined to obtain a first verification detection result. This first verification detection result is determined as the foreign object detection result for the area to be tested. The first verification detection result includes foreign object data points from the three-dimensional detection results and three-dimensional re-examined foreign object data points corresponding to the two-dimensional detection results. The three-dimensional re-examined foreign object data points represent foreign object data points present in the two-dimensional detection results but not present in the three-dimensional detection results; or... The two-dimensional detection result is re-examined to obtain a second verification detection result. The second verification detection result is determined as the foreign object detection result of the area to be tested. The second verification detection result includes foreign object data points in the two-dimensional AI detection result and two-dimensional AI re-examined foreign object data points. The two-dimensional AI re-examined foreign object data points represent other foreign object data points that are different from the foreign object data points in the two-dimensional AI detection result, and also represent other foreign object data points in the two-dimensional edge detection result that are different from the foreign object data points in the two-dimensional AI detection result.

2. The method according to claim 1, characterized in that, The three-dimensional detection results are re-examined to obtain a first verification detection result, including: The foreign object data points in the two-dimensional detection results are converted into three-dimensional foreign object data points; Determine the three-dimensional re-inspection foreign object data points, wherein the three-dimensional re-inspection foreign object data points are foreign object data points that are different from the foreign object data points in the three-dimensional detection results; The three-dimensional re-inspection foreign object data points are re-inspected to obtain the three-dimensional re-inspection results; The three-dimensional re-inspection results and the three-dimensional detection results are combined to obtain the first verification detection result.

3. The method according to claim 2, characterized in that, The method further includes: In response to the presence of foreign object data points in the three-dimensional re-inspection results, the first detection parameter is updated to the second detection parameter; The first detection parameter is a three-dimensional detection parameter used when detecting foreign objects in the area to be tested based on the three-dimensional point cloud data; the second detection parameter is a detection parameter used when re-inspecting the three-dimensional re-inspection foreign object data points. The parameter value corresponding to the first detection parameter is greater than the parameter value corresponding to the second detection parameter.

4. The method according to claim 1, characterized in that, The step of re-examining the two-dimensional detection result to obtain a second verification detection result includes: Convert foreign object data points in the 3D detection results into 2D foreign object data points; Determine the two-dimensional re-inspection foreign object data points, wherein the two-dimensional re-inspection foreign object data points are foreign object data points in the two-dimensional foreign object data points that are different from the foreign object data points in the two-dimensional AI detection results, and foreign object data points in the two-dimensional edge detection results that are different from the foreign object data points in the two-dimensional AI detection results; Based on an unsupervised detection network, the two-dimensional re-inspection foreign object data points are re-inspected to determine the two-dimensional AI re-inspection results; The second verification and detection results are obtained by merging the two-dimensional AI re-examination results and the two-dimensional AI detection results.

5. The method according to claim 4, characterized in that, The method further includes: In the second verification detection result, foreign object detection points for 2D AI detection are determined, and the foreign object detection points are added to the 2D AI training samples. The 2D AI training samples are used to train the 2D AI detection model, and the 2D AI detection model is used to determine the 2D AI detection result.

6. A foreign object detection device for a test area, characterized in that, include: The acquisition unit is used to acquire a three-dimensional image of the area to be measured using a three-dimensional image acquisition device, and to acquire three-dimensional point cloud data of the area to be measured based on the three-dimensional image; A conversion unit is used to determine the two-dimensional image data corresponding to the three-dimensional point cloud data based on the correspondence between the three-dimensional point cloud data and the two-dimensional image data, wherein the three-dimensional point cloud data includes the data points corresponding to each data point in the two-dimensional image data; The determining unit is used to perform foreign object detection on the area to be tested based on the three-dimensional point cloud data to obtain a three-dimensional detection result, and to perform foreign object detection on the area to be tested based on the two-dimensional image data to obtain a two-dimensional detection result, wherein the two-dimensional detection result includes a two-dimensional AI detection result and a two-dimensional edge detection result; The processing unit is used to perform a verification test on the three-dimensional detection result or the two-dimensional detection result, and determine the verification test result as the foreign object detection result of the area to be tested; The processing unit verifies the three-dimensional detection result or the two-dimensional detection result in the following manner: The three-dimensional detection results are re-examined to obtain a first verification detection result. This first verification detection result is determined as the foreign object detection result for the area to be tested. The first verification detection result includes foreign object data points from the three-dimensional detection results and three-dimensional re-examined foreign object data points corresponding to the two-dimensional detection results. The three-dimensional re-examined foreign object data points represent foreign object data points present in the two-dimensional detection results but not present in the three-dimensional detection results; or... The two-dimensional detection result is re-examined to obtain a second verification detection result. The second verification detection result is determined as the foreign object detection result of the area to be tested. The second verification detection result includes foreign object data points in the two-dimensional AI detection result and two-dimensional AI re-examined foreign object data points. The two-dimensional AI re-examined foreign object data points represent other foreign object data points that are different from the foreign object data points in the two-dimensional AI detection result, and also represent other foreign object data points in the two-dimensional edge detection result that are different from the foreign object data points in the two-dimensional AI detection result.

7. A foreign object detection system for a test area, characterized in that, include: A three-dimensional image acquisition device is used to acquire three-dimensional images of the area to be detected or measured. A foreign object detection device for a test area is used to acquire three-dimensional point cloud data of the test area based on the three-dimensional image, determine two-dimensional image data based on the correspondence between the three-dimensional point cloud data and the two-dimensional image data, perform foreign object detection on the test area based on the three-dimensional point cloud data to obtain a three-dimensional detection result, perform foreign object detection on the test area based on the two-dimensional image data to obtain a two-dimensional detection result, and determine the foreign object detection result of the test area based on the three-dimensional detection result and the two-dimensional detection result. The two-dimensional detection result includes two-dimensional AI detection result and two-dimensional edge detection result. The foreign object detection device for the area to be tested determines the foreign object detection result of the area to be tested based on the three-dimensional detection result and the two-dimensional detection result in the following manner: The three-dimensional detection results are re-examined to obtain a first verification detection result. This first verification detection result is determined as the foreign object detection result for the area to be tested. The first verification detection result includes foreign object data points from the three-dimensional detection results and three-dimensional re-examined foreign object data points corresponding to the two-dimensional detection results. The three-dimensional re-examined foreign object data points represent foreign object data points present in the two-dimensional detection results but not present in the three-dimensional detection results; or... The two-dimensional detection result is re-examined to obtain a second verification detection result. The second verification detection result is determined as the foreign object detection result of the area to be tested. The second verification detection result includes foreign object data points in the two-dimensional AI detection result and two-dimensional AI re-examined foreign object data points. The two-dimensional AI re-examined foreign object data points represent other foreign object data points that are different from the foreign object data points in the two-dimensional AI detection result, and also represent other foreign object data points in the two-dimensional edge detection result that are different from the foreign object data points in the two-dimensional AI detection result.

8. A foreign object detection device for a test area, characterized in that, include: processor: Memory used to store processor-executable instructions; The processor is configured to execute the foreign object detection method for the area to be tested according to any one of claims 1 to 5.

9. A storage medium, characterized in that, The storage medium stores instructions that, when executed by a processor, enable the processor to perform the foreign object detection method for the test area as described in any one of claims 1 to 5.

Images