Data correction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data correction method and device, electronic equipment and a storage medium. For 3D data with point cloud abnormity possibly caused by imaging abnormity, initial height data corresponding to a tin climbing area where abnormity possibly occurs are extracted according to surface structure characteristics of a to-be-detected object corresponding to the 3D data, and then according to the height distribution state of the tin climbing area, abnormal data are confirmed and collected from the initial height data. The method comprises the following steps: acquiring 3D data, removing acquired abnormal data to obtain effective height data with height missing, correcting the missing part based on the effective height data, and keeping the effective height in the 3D data to stabilize the performance of 3D data correction. The method can adapt to various detection targets to simply, quickly and accurately obtain correction results.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of optical detection technology, and in particular to a data correction method, device, electronic device and storage medium. Background Art

[0002] AOI (Automated Optical Inspection) is a technology based on optical principles that detects common defects encountered in welding production for various products involving welding processes, such as PCB (Printed Circuit Board) boards, to ensure product quality. During the AOI inspection process, it is necessary to first use the AOI device to photograph products that have been manually confirmed to meet production quality standards to obtain a standard inspection image, and then use the AOI device to photograph the product to be tested to obtain an image to be tested, compare the image to be tested with the inspection standard image, and confirm whether the product to be tested is qualified based on the comparison results.

[0003] Considering the limitations of two-dimensional detection in the vertical direction, in addition to capturing texture images with optical cameras during AOI detection, laser projection, structured light 3D sensors, etc. can also be used to obtain 3D height images of the product to be tested. However, the 3D height image may collapse or be lost to varying degrees due to imaging anomalies. Intuitively, the height data collected in some areas may not match the actual surface state of the product to be tested. In order to eliminate obvious data errors that occur during data collection, it can be repaired through height fusion and smoothing filtering.

[0004] The inventor studied the existing solution for repairing 3D height images in the AOI detection process and found that through the method of height fusion and smoothing filtering, there is a certain repair effect only for data anomalies in smaller areas; for data anomalies in larger areas, accurate repair cannot be completed simply and quickly. Summary of the invention

[0005] The present invention provides a data correction method, device, electronic device and storage medium to solve the technical problem that in the existing AOI detection process, when 3D height image data of a large area is abnormal, accurate repair cannot be completed simply and quickly.

[0006] In a first aspect, an embodiment of the present invention provides a data correction method, the data correction method comprising:

[0007] Obtain 3D data of the detection area corresponding to the component, the detection area includes the upper surface area of ​​the component, the tin creeping area and the surface area of ​​the printed circuit board, and extract the initial height data corresponding to the tin creeping area from the 3D data;

[0008] According to the height distribution law of the standard creep tin area, the effective height data is screened out from the initial height data according to the curvature information of each pixel point in the creep tin area;

[0009] The height data after correction of the creep area is determined based on the effective height data.

[0010] As mentioned above, for 3D data that may have point cloud anomalies due to abnormal imaging, the initial height data corresponding to the tin creeping area where the abnormality may occur is extracted according to the surface structure characteristics of the object to be detected corresponding to the 3D data, and then the abnormal data is confirmed to be collected from the initial height data according to the general height distribution of the tin creeping area, and then the abnormal data is removed to obtain the effective height data with height missing, wherein the missing part is the data in which the abnormality occurs during collection, and the missing part is corrected based on the effective height data. By retaining the effective height in the 3D data, the performance of 3D data correction is stable, and it can adapt to various detection targets to obtain correction results simply, quickly and accurately.

[0011] Among them, 3D data also includes texture data;

[0012] Accordingly, after determining the corrected height data of the creep area based on the effective height data, it also includes:

[0013] The corrected height data is fused with the texture data to obtain corrected 3D data, so as to perform tin creep defect detection based on the corrected 3D data.

[0014] As described above, the accuracy of defect detection can be improved based on the comprehensive judgment of height data and texture data.

[0015] Among them, according to the height distribution law of the standard creep tin area, the effective height data is screened out from the initial height data according to the curvature information of each pixel point in the creep tin area, including:

[0016] According to the height distribution corresponding to the tin creep area, confirm the collection of abnormal data from the initial height data;

[0017] Remove the abnormal data from the initial height data to obtain the valid height data.

[0018] As described above, by confirming the abnormal data from the initial height data and then removing the abnormal data, the height data that actually has defects and needs to be corrected can be obtained as accurately as possible.

[0019] Among them, according to the height distribution corresponding to the tin creep area, the abnormal data collected is confirmed from the initial height data, including:

[0020] Confirm the spatial curvature corresponding to each spatial point in the initial height data;

[0021] The point cloud data corresponding to the spatial points whose spatial curvature is within the preset abnormal curvature range are confirmed as the collected abnormal data.

[0022] As described above, based on the spatial curvature, the abnormal point cloud of the soldering area in the soldered components and printed circuit boards can be quickly identified.

[0023] Among them, the initial height data corresponding to the tin creeping area is extracted from the 3D data, including:

[0024] Confirming a first height parameter corresponding to an upper surface area of ​​a component and a second height parameter corresponding to a surface area of ​​a printed circuit board in the 3D data;

[0025] Confirming the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data according to the first height parameter and the second height parameter;

[0026] The height data outside the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data are extracted as the initial height data corresponding to the tin creeping area.

[0027] As described above, according to the structural characteristics of the soldered components and the printed circuit board, by dividing the 3D data with reference to three height ranges, it is possible to obtain a simple, fast and accurate correction result for the printed circuit board.

[0028] Wherein, confirming the first height parameter corresponding to the upper surface area of ​​the component and the second height parameter corresponding to the surface area of ​​the printed circuit board in the 3D data includes:

[0029] Sort all height values ​​in the 3D data from high to low;

[0030] Determine an average value of a preset proportion or a preset number of height values ​​that are ranked first as the first height parameter;

[0031] A preset second height parameter corresponding to a surface area of ​​the printed circuit board is obtained.

[0032] As described above, the top heights of the components are confirmed in a statistical manner according to the sizes of the components themselves, so as to realize rapid confirmation of the height variation area where the point cloud where data anomalies may occur is located.

[0033] Among them, 3D data also includes texture data;

[0034] Accordingly, the corrected height data of the creep area is obtained based on the effective height data, including:

[0035] The effective height data and texture data are input into the pre-trained height repair model to obtain the corrected height data of the creep area.

[0036] As mentioned above, by combining the pre-trained height restoration model with the height data and texture data for restoration, natural and accurate restoration can be achieved to obtain a result with a high degree of restoration.

[0037] In a second aspect, an embodiment of the present application provides a data correction device, the data correction device comprising:

[0038] The data acquisition unit is used to acquire 3D data of the detection area corresponding to the component, the detection area includes the upper surface area of ​​the component, the tin creeping area and the surface area of ​​the printed circuit board, and extract the initial height data corresponding to the tin creeping area from the 3D data;

[0039] A data screening unit is used to screen out effective height data from the initial height data according to the height distribution law of the standard creep tin area and the curvature information of each pixel point in the creep tin area;

[0040] The height repair unit is used to determine the corrected height data of the creep area based on the effective height data.

[0041] Wherein, the data correction device also includes:

[0042] The data fusion unit is used to fuse the corrected height data with the texture data to obtain corrected 3D data, so as to perform tin creep defect detection based on the corrected 3D data.

[0043] The data screening unit includes:

[0044] The abnormal confirmation module is used to confirm the collection of abnormal data from the initial height data according to the height distribution corresponding to the tin creep area;

[0045] The anomaly removal module is used to remove the collected anomaly data from the initial height data to obtain the effective height data.

[0046] Among them, the abnormal confirmation module includes:

[0047] The curvature confirmation submodule is used to confirm the spatial curvature corresponding to each spatial point in the initial height data;

[0048] The curvature comparison submodule is used to confirm the point cloud data corresponding to the spatial points whose spatial curvature is within the preset abnormal curvature range as the collected abnormal data.

[0049] The data screening unit includes:

[0050] A parameter confirmation module, used to confirm a first height parameter corresponding to an upper surface area of ​​a component and a second height parameter corresponding to a surface area of ​​a printed circuit board in the 3D data;

[0051] A point cloud confirmation module, used to confirm the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data according to the first height parameter and the second height parameter;

[0052] The height extraction module is used to extract the height data outside the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data as the initial height data corresponding to the tin creeping area.

[0053] Wherein, the first confirmation module includes:

[0054] The height sorting submodule is used to sort all height values ​​in the 3D data from high to low;

[0055] A first confirmation submodule is used to confirm an average value of a preset proportion or a preset number of height values ​​that are ranked top as a first height parameter;

[0056] The second confirmation submodule is used to obtain a preset second height parameter corresponding to the surface area of ​​the printed circuit board.

[0057] Among them, 3D data also includes texture data;

[0058] Accordingly, the high-level repair unit includes:

[0059] The prediction correction module is used to input the effective height data and texture data into the pre-trained height repair model to obtain the corrected height data of the creep area.

[0060] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0061] one or more processors;

[0062] a memory for storing one or more computer programs;

[0063] When one or more computer programs are executed by one or more processors, the electronic device implements the data correction method of the first aspect.

[0064] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data correction method of the first aspect.

[0065] The electronic device and computer of the third and fourth aspects can be used to execute the data correction method provided by any of the above embodiments, and have corresponding functions and beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A method flow chart of a data correction method provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of the effect of 3D imaging of 3D data to be processed;

[0069] Figure 3 A schematic diagram of the structure of a height repair model provided in an embodiment of the present application;

[0070] Figure 4 is a schematic diagram of a texture image of an object to be detected;

[0071] Figure 5 for Figure 3 A schematic diagram of the upper surface area of ​​a component of the inspection object in the texture image;

[0072] Figure 6 for Figure 3 A schematic diagram of a height variation region of a detected object in a texture image is shown;

[0073] Figure 7 for Figure 3 A schematic diagram of a surface area of ​​a printed circuit board of a detection object in a texture image;

[0074] Figure 8 It is a schematic diagram of the plane effect of projecting the 3D data to be processed parallel to the height direction;

[0075] Fig. 9 A schematic diagram of the regional distribution confirmed for the 3D data to be processed;

[0076] Fig.10 for Fig. 9 A schematic diagram of the distribution of effective height data corresponding to the 3D data to be processed;

[0077] Fig.11 Based on Fig.10 A schematic diagram showing the distribution of correction result data obtained from the effective height data shown;

[0078] Fig.12 Based on Fig.11 The calibration results shown are Fig.10 Schematic diagram of the distribution of the resulting 3D data obtained from the effective height data shown;

[0079] Fig.13 A schematic diagram of the structure of a data correction device provided in an embodiment of the present application;

[0080] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are used to explain the present invention, rather than to limit the present invention. It should also be noted that, for the sake of ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0082] It should be noted that due to space limitations, this application description does not list all optional implementation methods. After reading this application description, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0083] Each embodiment is described in detail below.

[0084] During the 3D AOI shooting process, imaging anomalies usually cause the 3D height image to collapse or lose to varying degrees, and are more obvious in the fusion welding connection area of ​​products produced by welding. For example, in the circuit board production process, it is more obvious in the tin creeping area. Figure 2 The figure shows a schematic diagram of the effect of 3D imaging of the 3D data to be processed. Figure 2 An imaging anomaly of a tin creep area is shown in FIG. Figure 2 The tin creep area in the image shows an abnormal collapse. This area contains information about tin creep (tin paste amount, pad offset, etc.), which plays an important guiding role in whether the tin creep target detection is normal.

[0085] For imaging anomalies in these locations, if they are minor, such as pixel areas smaller than 7*7, they can be repaired through high-degree fusion and smoothing filtering by multi-cameras. However, for imaging anomalies in larger areas, if high-degree fusion and smoothing filtering are used, the adjacent pixel height information is used to complete the repair. Multiple iterations are required for repairing larger areas, which will cause the details of the tin creeping area to be blurred and feature loss to occur. Multiple iterations of the entire image will also result in additional resource consumption.

[0086] Using a deep learning-based segmentation model is also a way to implement repair, but this method requires training on multiple annotated tin creeping pictures, which will consume a lot of manual annotation resources and is difficult to generalize to new scenes. The effect of the segmentation model also directly affects the subsequent completion, bringing uncertainty to the tin creeping completion effect. Moreover, this method is to filter out all tin creeping areas (including areas where normal data is collected) and start to complete from the height blank of the entire tin creeping area. When there are large gaps in the point cloud data, it is difficult for the segmentation model to infer the missing tin creeping height from texture information alone.

[0087] In general, there is currently no simple, quick, and accurate repair solution to solve abnormal situations such as height deviation and loss in large areas.

[0088] In order to solve the above technical problems, the embodiment of the present application proposes a data correction method. For 3D data that may have point cloud anomalies due to imaging abnormalities, the initial height data corresponding to the tin creeping area where the abnormality may occur is extracted according to the surface structure characteristics of the object to be detected corresponding to the 3D data, and then the collected abnormal data is confirmed from the initial height data according to the general height state of the tin creeping area, and then the collected abnormal data is removed to obtain effective height data with height missing, wherein the missing data is the data in which the abnormality occurs during the collection, and the missing part is corrected based on the effective height data. By retaining the effective height in the 3D data, the performance of 3D data correction is stable, and it can adapt to various detection targets to obtain correction results simply, quickly and accurately.

[0089] Figure 1 is a method flow chart of a data correction method provided by an embodiment of the present application, such as Figure 1 As shown, the data correction method includes but is not limited to steps S110 to S130.

[0090] Step S110: Acquire 3D data of the detection area corresponding to the component, the detection area includes the upper surface area of ​​the component, the tin creeping area and the surface area of ​​the printed circuit board, and extract the initial height data corresponding to the tin creeping area from the 3D data.

[0091] 3D data is an image obtained by collecting point clouds of objects to be inspected during automatic optical inspection using image acquisition methods such as optical cameras (multiple), laser projection, and structured light 3D sensors. It describes the surface shape of the object to be inspected, that is, the height values ​​of multiple discrete spatial points are used to represent the surface fluctuations of the object to be inspected. The data corresponding to each spatial point includes two digital position coordinates and a digital height value. Specific objects to be inspected include welding of various metal parts, soldering of printed circuit boards, etc.

[0092] Step S120: According to the height distribution rule of the standard solder creeping area, effective height data is screened out from the initial height data according to the curvature information of each pixel point in the solder creeping area.

[0093] When collecting 3D data of the object to be detected, the planar structure area parallel to the collection surface usually does not have collection anomalies. Collection anomalies basically occur when collecting structural areas with large changes in distance from the collection equipment (such as the creeping tin area formed when components are soldered to the printed circuit board). Therefore, in the embodiment of the present application, only the point cloud data corresponding to the structural area with more complex surface shape is processed in a targeted manner, that is, the height data of the surface area of ​​the printed circuit board and the upper surface area of ​​the components are extracted from the 3D data. Considering that this area presents changes in height values ​​in the 3D data, and most of them also have undulating surface shapes, the creeping tin area can also be defined as a height change area based on the height change characteristics of the data level.

[0094] The identification of height change areas can be directly confirmed by the height of each spatial point. The overall idea is that a series of spatial points whose height values ​​of adjacent spatial points have been changing or can be regarded as changing (for example, a series of spatial points whose heights remain unchanged for a small number of consecutive point clouds and then continue to change) are all regarded as spatial points in the height change area, and the height data corresponding to the height change area is the initial height data. It should be understood that the height change area only needs to have fluctuations in the height data due to the existence of image acquisition anomalies, and is not limited to continuous rise or fall changes.

[0095] In another optional implementation, based on the basic composition characteristics of the circuit board surface, that is, the printed circuit board surface area with basically consistent height, the component upper surface area with basically consistent height, and the tin creeping area with varying height, the printed circuit board surface area and the component upper surface area can be first confirmed according to the height relationship, and then the remaining part is the tin creeping area, that is, confirm the first height parameter corresponding to the component upper surface area and the second height parameter corresponding to the printed circuit board surface area in the 3D data; confirm the component upper surface area and the printed circuit board surface area in the 3D data according to the first height parameter and the second height parameter; extract the height data outside the component upper surface area and the printed circuit board surface area in the 3D data as the initial height data.

[0096] For industrial production scenarios using AOI, the connections formed by various devices and processing links (such as tin creep) have corresponding relatively strict standards, that is, the height change area has a corresponding uniform surface state. For example, tin creep is formed by the solidification of the solder paste overflowed during tin welding. After the solder paste overflows and solidifies, the surface usually presents a relatively smooth convex surface. This convex smoothing effect should show that the spatial curvature of the corresponding spatial points is greater than 0 in the collected 3D data. If the data of a certain area of ​​tin creep is abnormal, the spatial curvature of the corresponding spatial point will appear less than 0. Therefore, the height data whose spatial curvature does not conform to the conventional change rules can be confirmed as the collected abnormal data. Of course, the corresponding height change rules can also be preset according to the product type and shape characteristics of the specific object to be detected to confirm the collection of abnormal data. If no abnormal data is collected, it can be considered that the 3D data has no data abnormality at least in the image acquisition link, and the specific defect detection can be performed directly with the 3D data and the corresponding texture image.

[0097] For 3D data, after removing the abnormal data, what is retained is the effective information containing all the real height values, which can provide a positive impact on the correction. Therefore, the data remaining after removing the abnormal data is used as the effective height data. Subsequently, the effective height data that is missing due to the removal of the abnormal data is repaired for the lost height values.

[0098] When removing abnormal data, we can confirm whether a single height data in the initial height data is abnormal data according to the relationship between the normal vector corresponding to the initial height data and the camera point. That is, when judging the abnormal data, we combine the point cloud data and the image features captured by the camera to make a judgment. Because the creeping area is blocked, there will be some areas with abnormal imaging. Combined with the camera shooting, if the normal vector of a creeping point is near the camera point, then the probability that this point is the correct height is relatively high, and the corresponding data is normal data, otherwise it is abnormal data.

[0099] Step S130: determining the corrected height data of the creep area based on the effective height data.

[0100] A specific method for correcting the effective height data is to input the effective height data and the texture image corresponding to the object to be detected into a pre-trained height repair model to obtain the corrected height data of the tin creeping area.

[0101] The height restoration model can be trained based on various network models, such as a network model based on deep learning. The height restoration model can be trained specifically in the following ways: obtain standard 3D data and the original texture image corresponding to the same object as the standard 3D data; perform missing processing on the spatial points corresponding to the height change area in the standard 3D data to obtain corrected sample data with blank spatial points in the height change area; combine each corrected sample data and the corresponding original texture image and input it into a network model based on deep learning, train it with the corresponding standard 3D data as the output target, and obtain the height restoration model.

[0102] In the case of the training sample design and training objectives described above, the specific process of training the network model can refer to the conventional implementation of neural network model training, and the specific prediction stage corresponds to the implementation principle of the training stage. Figure 3 A specific implementation method shown in the figure adopts an encoder-decoder structure 32, which includes a feature processing unit 33 composed of multiple data processing layers. The two types of data composed of texture images and incomplete data are input to the height restoration model through the input layer 31. If in the training stage, the incomplete data is specifically the correction sample data, the training goal is that the complete data output by the output layer 34 can be the same as the corresponding standard 3D data. Of course, in the specific implementation, the complete data output by the output layer 34 is the same as the standard 3D data. The training can be terminated when the similarity expected by the user is actually achieved. If in the prediction stage, the incomplete data is specifically the effective height data, and the corresponding complete data is the correction result data, that is, the height data after correction corresponding to the entire tin crawling area.

[0103] It should be noted that inputting the effective height data and the texture image corresponding to the object to be detected into the pre-trained height repair model to obtain the height data after correction of the tin creeping area is only an optional implementation method. In the specific implementation process, if the height repair model is trained only based on the height data, then during data correction, only the effective height data can be input into the pre-trained height repair model to obtain the height data after correction of the tin creeping area.

[0104] In another optional implementation, the normal point cloud data may be confirmed based on the effective height data, and the removed part of the data may be supplemented by interpolation to obtain complete correction result data.

[0105] For the initially collected 3D data, it may include collected normal data and collected abnormal data, wherein the collected normal data is the basis for predicting the collected abnormal data, and is also accurate and usable data, and can be directly retained in the end; the collected abnormal data is removed from the beginning, and the correction result of the collected abnormal data is obtained by predicting the collected normal data in the subsequent process. In the embodiment of the present application, the correction of the collected abnormal data does not require the participation of the collected abnormal data itself, and only needs to use the correction result data output by the height repair model (or the correction result data obtained by interpolation and other methods) directly as the height data after correction of the tin creeping area, that is, to obtain the corrected 3D data (especially including the height data obtained by correcting the tin creeping area), and then use the corrected 3D data and the corresponding texture image for specific defect detection. Specifically, the corrected height data can be fused with the texture data to obtain the corrected 3D data, so as to reconstruct the image of components based on the corrected 3D data, or perform tin creeping defect detection.

[0106] Based on the above overall implementation process, the following Figures 4 to 12 , based on the automatic optical inspection scenario of circuit boards, an exemplary description of the embodiments of the present application is given.

[0107] A circuit board is the state after components and printed circuit boards are soldered together. Compared with electronic devices made of circuit boards, circuit boards can be regarded as combined parts that have just completed certain assembly steps. When performing AOI inspection on a circuit board, the AOI equipment collects 3D data (i.e., 3D data to be processed) and also collects texture images. Texture images are usually obtained by placing an optical camera vertically downward toward a storage table, and the circuit board is placed flat on the storage table to collect the texture image. The texture image is actually an optical image. The components in the circuit board are located on the side facing away from the storage table. Here, the side where the components are located is defined as the upper side. Corresponding to a single component in the circuit board, it appears as follows in the texture image. Figure 4 The top view shown in the figure shows the effect of Figure 4 In the figure, the top of the component has the word "104". The upper side of the circuit board can be roughly divided into three surface states, namely the upper surface of the component at the top, the uncovered area of ​​the printed circuit board at the bottom, and the area between the two. The area between the two includes the side surfaces of various components, the tin creep formed when the components are welded to the printed circuit board, etc.

[0108] for Figure 4For the components installed on the printed circuit board shown in the figure, the upper surface of the component and the uncovered area of ​​the bottommost printed circuit board can be considered to be in two different planes. Correspondingly, when confirming the initial area point cloud data, the first height parameter corresponding to the top of the component and the second height parameter corresponding to the surface of the printed circuit board in the 3D data to be processed can be confirmed first; then, the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data to be processed are confirmed based on the first height parameter and the second height parameter; the height data outside the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data to be processed are extracted as the initial height data corresponding to the tin creeping area. In this processing process, it is equivalent to Figure 4 The 3D data to be processed corresponding to the components shown is considered to be composed of three parts, namely, the top of the component (i.e. Figure 5 The point cloud data corresponding to the spatial points on the surface of the printed circuit board (i.e., the light-colored component upper surface area near the center) corresponds to the point cloud data corresponding to the spatial points on the surface of the printed circuit board (i.e., Figure 7 The point cloud data corresponding to the spatial points of the printed circuit board surface area covered by the light color in the middle and outer periphery, and the remaining points between the two (i.e. Figure 6 The point cloud data corresponding to the spatial points (the height change areas covered by light colors at both ends of the components). The top of the component and the surface of the printed circuit can both be regarded as planes, and the height values ​​of the spatial points at the same position are basically the same (that is, the first height parameter corresponding to the top of the component and the second height parameter corresponding to the surface of the printed circuit board). Taking into account the fluctuations of data during actual detection, for printed circuit boards with small individual thickness differences, the second height parameter is set to have a certain height, and is exemplarily set to 2 to 5μm. For example, the height value below 3μm is the height value of the spatial point located on the printed circuit board. For components with large individual differences, the first height parameter can be confirmed by statistics. On the whole, the height value above the first height parameter is the height value of the spatial point located at the top of the component, and the height value between the first height parameter and the second height parameter is the height value corresponding to the spatial point in the height change area.

[0109] When confirming the first height parameter by statistics, all height values ​​in the 3D data to be processed can be sorted from high to low first; the average value of a preset proportion or a preset number of height values ​​that are ranked top is confirmed as the first height parameter; and a preset second height parameter corresponding to the surface of the printed circuit board is obtained. Figure 5 , Figure 6 and Figure 7 The light-colored coverage in the figure can be defined as a mask, and the corresponding three masks are represented as the upper surface area of ​​the component Mask Chip , Height Change Area Mask Soder Mask the printed circuit board surface area Boerd, the height values ​​corresponding to the spatial points in each area are Height Chip 、Height Solder With Helght Board , sort the height values ​​of all spatial points in the 3D data to be processed from high to low, and take the average value of the preset proportion or number of height values ​​in the sorted height values avg The preset ratio is, for example, a value between 1 / 10 and 1 / 5, and 1 / 6 is taken as an example; the preset number is, for example, a value between 50 and 100, and 60 is taken as an example. According to the height distribution relationship Height determined by the regional relationship Chip >Height avg >Helght Solder , you can quickly confirm the surface area Mask of the component Chip Based on the definition and value setting of the second height parameter in the previous article, the height distribution relationship Height can be obtained Solder >3μm>Helght Board , you can confirm the printed circuit board surface area Mask Boerd , remove the mask of the upper surface area of ​​the component from the 3D data to be processed Chip and PCB surface area Mask Board After that, the remaining area is the height change area Mask Solder Of course, the average value avg It is an estimated value and may not be the height of the top surface of the component. However, in the AOI inspection of the circuit board, the inspection of components is mainly the inspection of the welding effect. A single component itself can usually be inspected for defects without AOI. The creep formed by welding usually does not cover the entire side of the component. And the average value of the height value at the front is obtained after sorting from high to low. avg Even if it is not the top surface of the component, it can be guaranteed to be close to the top surface of the component, without affecting the entire tin climbing to be kept in the height change area Mask Solder , which can ensure that all abnormal data collected that may occur during tin creep are corrected.

[0110] In the embodiment of the present application, the height change area Mask SalderThe corresponding point cloud data is defined as the initial area point cloud data. The collected abnormal data in the initial area point cloud data is usually a height collapse or loss at certain spatial points. At this time, the spatial curvature corresponding to each spatial point in the initial area point cloud data can be confirmed first; then the point cloud data corresponding to the spatial point whose spatial curvature is within the preset abnormal curvature range is confirmed as the collected abnormal data, and then the collected abnormal data can be filtered out. In the specific processing process, the abnormal curvature range is a preset static value (such as 0) or a received input value. The input value means that when detection is required, the specific parameter value of the abnormal area range can be adjusted according to the corresponding relationship between the actual curvature of the specific circuit board and the normal spatial point, and then the collected abnormal data is confirmed and filtered out, and the normal data is retained as a clue for correction. The spatial area can be confirmed by the second-order derivative. The height data corresponding to each point cloud in the initial area point cloud data is the initial height data.

[0111] For example Figure 8 This is a schematic diagram of the plane effect of projecting a 3D data 1 parallel to the height direction. In the case where the area is not divided based on the height value, the point cloud data corresponding to each spatial point in the 3D data 1 is relatively equal and discrete, that is, Figure 8 There is no special spatial point as shown. After confirming the first height parameter and the second height parameter, the following can be obtained: Fig. 9 As shown in the regional distribution diagram, the entire 3D data 1 can be divided into the upper surface area 13 of the component, the creeping tin area 12 and the printed circuit board surface area 11. By identifying the abnormal data collected in the creeping tin area 12 through the spatial curvature, the following can be obtained: Fig.10 The point cloud data of the space points corresponding to the first sub-area 121 and the second sub-area 122 shown are collected abnormal data. Fig.11 As shown, the creeping tin area 12 removes the collected abnormal data of the first sub-area 121 and the second sub-area 122 to obtain effective height data. In the effective height data, the height values ​​of the corresponding spatial points of the first sub-area 121 and the second sub-area 122 are blank. The effective height data is combined with the corresponding texture image 12' and input into the height repair model 3. The height repair model 3 outputs the correction result data corresponding to the creeping tin area 12. In the correction result data, the first sub-area 121 and the second sub-area 122 present new height values, and the height repair model 3 may repair the height values ​​in a larger range with the first sub-area 121 and the second sub-area 122 as the center during the prediction and repair process, that is, Fig.11The point cloud data of the space points corresponding to the first correction sub-area 121 'and the second correction sub-area 122 'as shown are all repaired. In the embodiment of the present application, the complete repair result of the height of the creeping tin area can be directly used as the final height data. In subsequent inspections, the original component upper surface area 13 and the printed circuit board surface area 11, as well as the height data of the corrected creeping tin area 12 are combined, and optical inspection is completed on the basis of the combination, such as optical inspection of the creeping tin area.

[0112] If the existing technology is used to process the same 3D data of the circuit board, the creeping area can be repaired by Gaussian / smoothing filtering based on the height distribution between adjacent pixels, but this processing method is prone to blurring the height features. A completion model can also be used, but the completion model depends on the performance of the segmentation model. For data from different manufacturers, the segmentation may fail because it has not been trained specifically, and its generalization ability cannot meet production requirements; in addition, the strategy of starting from a blank height map is difficult to effectively complete the original height, and its instability also cannot meet the use requirements.

[0113] The data correction method corresponding to the circuit board provided in the embodiment of the present application accurately segments the area of ​​tin creeping according to the height filtering of the components. Combined with the curvature distribution of the abnormal points in the tin creeping area, the effective height is retained as a clue, which plays an important guiding role in height completion. In addition, a repair network is constructed based on deep learning, and the lack of height is simulated during training, and the repaired height map is finally output. The processing accuracy is high, with good stability and generalization ability, which can effectively improve the quality of imaging, restore the true characteristics of the tin creeping height, and effectively improve the stability of AOI equipment detection.

[0114] In the above data correction method, for 3D data that may have point cloud anomalies due to abnormal imaging, the initial height data corresponding to the tin creeping area where the abnormality may occur is extracted based on the surface structure characteristics of the object to be detected corresponding to the 3D data, and then the abnormal data is confirmed to be collected from the initial height data based on the general height state of the tin creeping area, and then the abnormal data is removed to obtain valid height data with height missing, wherein the missing part is the data in which the abnormality occurs during collection, and the missing part is corrected based on the valid height data. By retaining the effective height in the 3D data, the performance of 3D data correction is stable, and it can adapt to various detection targets to obtain correction results simply, quickly and accurately.

[0115] Fig.13 This is a schematic diagram of the structure of a data correction device provided in an embodiment of the present application. Fig.13 As shown, the data correction device includes: a data acquisition unit 210, a data screening unit 220 and a height repair unit 230.

[0116] Among them, the data acquisition unit 210 is used to obtain 3D data of the detection area corresponding to the component, the detection area includes the upper surface area of ​​the component, the tin creeping area and the surface area of ​​the printed circuit board, and extract the initial height data corresponding to the tin creeping area from the 3D data; the data screening unit 220 is used to filter out the effective height data from the initial height data according to the height distribution law of the standard tin creeping area and the curvature information of each pixel point in the tin creeping area; the height repair unit 230 is used to determine the corrected height data of the tin creeping area based on the effective height data.

[0117] Based on the above embodiment, the data correction device further includes:

[0118] The data fusion unit is used to fuse the corrected height data with the texture data to obtain corrected 3D data, so as to perform tin creep defect detection based on the corrected 3D data.

[0119] Based on the above embodiment, the height repair unit 230 includes:

[0120] The abnormal confirmation module is used to confirm the collection of abnormal data from the initial height data according to the height distribution corresponding to the tin creep area;

[0121] The anomaly removal module is used to remove the collected anomaly data from the initial height data to obtain the effective height data.

[0122] Based on the above embodiment, the abnormality confirmation module includes:

[0123] The curvature confirmation submodule is used to confirm the spatial curvature corresponding to each spatial point in the initial height data;

[0124] The curvature comparison submodule is used to confirm the point cloud data corresponding to the spatial points whose spatial curvature is within the preset abnormal curvature range as the collected abnormal data.

[0125] Based on the above embodiment, the data screening unit 220 includes:

[0126] A parameter confirmation module, used to confirm a first height parameter corresponding to an upper surface area of ​​a component and a second height parameter corresponding to a surface area of ​​a printed circuit board in the 3D data;

[0127] A point cloud confirmation module, used to confirm the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data according to the first height parameter and the second height parameter;

[0128] The height extraction module is used to extract the height data outside the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data as the initial height data corresponding to the tin creeping area.

[0129] Based on the above embodiment, the first confirmation module includes:

[0130] The height sorting submodule is used to sort all height values ​​in the 3D data from high to low;

[0131] A first confirmation submodule is used to confirm an average value of a preset proportion or a preset number of height values ​​that are ranked top as a first height parameter;

[0132] The second confirmation submodule is used to obtain a preset second height parameter corresponding to the surface area of ​​the printed circuit board.

[0133] Based on the above embodiment, the 3D data also includes texture data;

[0134] Accordingly, the height repair unit 230 includes:

[0135] The prediction correction module is used to input the effective height data and texture data into the pre-trained height repair model to obtain the corrected height data of the creep area.

[0136] The data correction device provided in the embodiment of the present application is included in an electronic device and can be used to execute the corresponding data correction method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0137] It is worth noting that in the embodiment of the above-mentioned data correction device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0138] Fig.14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.14 As shown, the electronic device includes a processor 310 and a memory 320. The interactive tablet and conference control device mentioned above are both electronic devices in terms of hardware architecture. In a common electronic device product form, an input device 330, an output device 340 and a communication device 350 may also be included. The number of processors 310 in the electronic device may be one or more. Fig.14 A processor 310 is taken as an example; the processor 310, the memory 320, the input device 330, the output device 340 and the communication device 350 in the electronic device can be connected via a bus or other means. Fig.14 The example of connecting through bus is taken in the following.

[0139] The memory 320 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the data correction method in the embodiment of the present application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 320, that is, implements the above-mentioned data correction method.

[0140] The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0141] The input device 330 may be used to receive network configuration information. The output device 340 may include a display device such as a display screen.

[0142] The above electronic device can be used to execute any data correction method and has corresponding functions and beneficial effects.

[0143] An embodiment of the present invention also provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to perform relevant operations in the data correction method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0144] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0145] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0149] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data correction method, characterized in that: include: Acquire 3D data of a detection area corresponding to a component, wherein the detection area includes an upper surface area of ​​the component, a tin creeping area, and a surface area of ​​a printed circuit board, and extract initial height data corresponding to the tin creeping area from the 3D data; According to the height distribution law of the standard tin creeping area, the effective height data is screened out from the initial height data according to the curvature information of each pixel point in the tin creeping area; The corrected height data of the solder creep area is determined based on the effective height data.

2. The data correction method according to claim 1, characterized in that: The 3D data also includes texture data; Correspondingly, after determining the height data of the tin creep area after correction based on the effective height data, the method further includes: The corrected height data is fused with the texture data to obtain corrected 3D data, so as to perform tin creep defect detection based on the corrected 3D data.

3. The data correction method according to claim 1, characterized in that: The method of filtering out effective height data from the initial height data according to the height distribution law of the standard creep tin area and the curvature information of each pixel point in the creep tin area includes: According to the height distribution corresponding to the tin creeping area, confirming the collection of abnormal data from the initial height data; The abnormal acquisition data is removed from the initial height data to obtain effective height data.

4. The data correction method according to claim 3, characterized in that: The step of confirming and collecting abnormal data from the initial height data according to the height distribution corresponding to the tin creeping area includes: Determining the spatial curvature corresponding to each spatial point in the initial height data; The point cloud data corresponding to the space points whose space curvature is within a preset abnormal curvature range are confirmed as the collected abnormal data.

5. The data correction method according to claim 1, characterized in that: Extracting the initial height data corresponding to the tin creeping area from the 3D data includes: Confirming a first height parameter corresponding to an upper surface area of ​​the component and a second height parameter corresponding to a surface area of ​​the printed circuit board in the 3D data; Confirming the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data according to the first height parameter and the second height parameter; Height data outside the upper surface area of ​​the component and the surface area of ​​the printed circuit board in the 3D data are extracted as initial height data corresponding to the tin creeping area.

6. The data correction method according to claim 5, characterized in that: The confirming of the first height parameter corresponding to the upper surface area of ​​the component and the second height parameter corresponding to the surface area of ​​the printed circuit board in the 3D data includes: Sort all height values ​​in the 3D data from high to low; Determine an average value of a preset proportion or a preset number of height values ​​that are ranked top, as the first height parameter; A preset second height parameter corresponding to the surface area of ​​the printed circuit board is obtained.

7. The data correction method according to claim 3, characterized in that: The 3D data also includes texture data; Accordingly, obtaining the corrected height data of the tin creeping area based on the effective height data includes: The effective height data and the texture data are input into a pre-trained height repair model to obtain the corrected height data of the tin creeping area.

8. A data correction device, characterized in that: include: A data acquisition unit, used to acquire 3D data of a detection area corresponding to a component, wherein the detection area includes an upper surface area of ​​the component, a tin creeping area, and a surface area of ​​a printed circuit board, and extract initial height data corresponding to the tin creeping area from the 3D data; A data screening unit, used for screening out effective height data from the initial height data according to the height distribution law of the standard creep tin area and the curvature information of each pixel point in the creep tin area; The height repair unit is used to determine the corrected height data of the tin creep area based on the effective height data.

9. An electronic device, characterized in that include: one or more processors; a memory for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the data correction method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data correction method as described in any one of claims 1 to 7 is implemented.

Images