Image correction method and device, computer equipment and storage medium
By using feature point matching and clustering techniques, the problem of insufficient accuracy of existing image correction methods in industrial inspection is solved, achieving stable and reliable image correction and improving the accuracy of detection and recognition.
Patent Information
- Application Number
- CN202410635406.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-21
- Publication Date
- 2024-11-08
AI Technical Summary
Existing image correction methods based on Hough transform and deep convolutional neural networks are not accurate enough in handling image tilt and perspective distortion in industrial inspection, and they have high data requirements, poor generalization ability, and cannot adapt to image correction in various situations.
The image is corrected by matching the feature point sets of the image to be detected and the reference image. If the number of matching feature points is insufficient, the distribution of feature points on the reference image is determined, and clustering and feature point pairing are performed. The affine transformation matrix is calculated using multiple feature point matching pairs.
It improves the stability and reliability of image correction, provides more accurate image input, and is suitable for image correction in industrial inspection scenarios such as printed circuit boards, thereby enhancing the accuracy of detection and recognition.
Smart Images

Figure CN118918048A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image correction method, device, computer equipment and storage medium. Background Art
[0002] Computer vision is increasingly used in industrial inspection. When acquiring the image to be inspected, there is a deviation between the industrial camera and the object to be inspected, which causes the image to be inspected to be tilted. Therefore, it is necessary to correct the image to be inspected to improve the accuracy of the inspection.
[0003] In the related art, a method based on Hough transform is used to correct the image to be detected. However, the accuracy of this method needs to be improved. Summary of the invention
[0004] The embodiments of this specification are intended to solve at least one of the technical problems in the related art to a certain extent. To this end, the embodiments of this specification propose an image correction method, device, computer equipment and storage medium.
[0005] An embodiment of the present specification provides an image correction method, the method comprising:
[0006] Acquire a first feature point set of the image to be detected and perform feature point matching with a second feature point set of the reference image to obtain target feature points in the first feature point set that match the second feature point set;
[0007] If the number of the target feature points is less than a preset number threshold, determining the distribution of the second feature points in the second feature point set on the reference image;
[0008] Clustering and feature point pairing are performed according to the distribution, the first feature point set, and the second feature point set to obtain a plurality of feature point matching pairs; wherein the number of the feature point matching pairs is greater than the preset number threshold;
[0009] The image to be detected is corrected based on the matching of the multiple feature points to obtain a corrected detection image.
[0010] In one embodiment, clustering and pairing feature points according to the distribution, the first feature point set, and the second feature point set to obtain a plurality of feature point matching pairs includes:
[0011] Determine the number of clusters according to the distribution;
[0012] Clustering the first feature point set according to the number of clusters to obtain a plurality of clusters;
[0013] Each cluster is used to pair feature points with feature points in the second feature point set to obtain the plurality of feature point matching pairs.
[0014] In one embodiment, each cluster has an initial matching point in the second feature point set; and the method of pairing feature points of each cluster with feature points in the second feature point set to obtain the plurality of feature point matching pairs includes:
[0015] Each cluster is matched with an initial matching point corresponding to each cluster to obtain the plurality of feature point matching pairs.
[0016] In one embodiment, the first feature point set includes first feature points obtained by performing target detection on the image to be detected; and matching each cluster with an initial matching point corresponding to each cluster to obtain the plurality of feature point matching pairs includes:
[0017] If any cluster is not empty, matching is performed based on the distance data between the first feature point in any cluster and the initial matching point to obtain a feature point matching pair of at least part of the first feature points in any cluster.
[0018] In one embodiment, the first feature point set includes first feature points obtained by performing target detection on the image to be detected; and the matching of each cluster with the initial matching point corresponding to each cluster to obtain the plurality of feature point matching pairs includes at least one of the following situations:
[0019] If any cluster is empty, no feature point matching pair is generated;
[0020] If there is a first feature point in any cluster, a matching feature point matching the first feature point is determined from the initial matching points corresponding to the cluster, and a feature point matching pair is formed accordingly;
[0021] If there are multiple first feature points in any cluster, and any cluster corresponds to an initial matching point, a first feature point matching the initial matching point is determined from the multiple first feature points, and a feature point matching pair is formed accordingly;
[0022] If there are multiple first feature points in any cluster, and any cluster corresponds to multiple initial matching points, for any first feature point in any cluster, the matching feature points that match the any first feature point are determined from the initial matching points corresponding to the any cluster, and the corresponding feature point matching pairs constitute the any first feature point.
[0023] In one embodiment, the correcting the image to be detected based on the matching of the multiple feature points to obtain a corrected detection image includes:
[0024] Performing affine transformation based on the plurality of feature point matching pairs to obtain an affine transformation matrix;
[0025] The image to be detected is corrected using the affine transformation matrix to obtain a corrected detection image.
[0026] In one embodiment, the first feature point set is obtained by:
[0027] Performing target detection on the image to be detected to obtain position data of multiple detection frames;
[0028] Determine the center point of each detection frame based on the position data of each detection frame;
[0029] The first feature point set is formed based on the center point of each detection frame.
[0030] In one embodiment, the target feature point is determined by:
[0031] Correcting the first feature point set according to a coordinate deviation between a first feature point in the first feature point set and a second feature point in the second feature point set to obtain a corrected feature point set;
[0032] Determine, based on the distance data between the third feature point and the second feature point in the corrected feature point set, the feature points to be removed that meet the preset distance screening condition in the corrected feature point set;
[0033] According to the revised feature point set and the second feature point set, it is determined whether to delete the feature points to be removed from the revised feature point set, so as to determine the target feature points that need to be retained in the first feature point set.
[0034] In one embodiment, determining whether to delete the feature points to be removed from the revised feature point set based on the revised feature point set and the second feature point set includes:
[0035] Try to delete the feature points to be removed from the corrected feature point set to obtain a post-deletion feature point set;
[0036] Determining, in the second feature point set, verification feature points corresponding to each feature point in the deleted feature point set to obtain a verification feature point set;
[0037] If the sum of the distance deviations between the deleted feature point set and the verified feature point set is less than the deviation and threshold, determining to delete the feature point to be removed;
[0038] If the sum of the distance deviations between the deleted feature point set and the verified feature point set is not less than the deviation sum threshold, it is determined that the feature point to be removed is not deleted.
[0039] The present specification provides an image correction device, the device comprising:
[0040] A feature point matching module is used to obtain a first feature point set of the image to be detected and a second feature point set of the reference image to perform feature point matching to obtain target feature points in the first feature point set that match the second feature point set;
[0041] a distribution determination module, configured to determine the distribution of the second feature points in the second feature point set on the reference image if the number of the target feature points is less than a preset number threshold;
[0042] A feature point pairing module, configured to perform clustering and feature point pairing according to the distribution, the first feature point set, and the second feature point set to obtain a plurality of feature point matching pairs; wherein the number of the feature point matching pairs is greater than the preset number threshold;
[0043] The image correction module to be detected is used to correct the image to be detected based on the matching of the multiple feature points to obtain a corrected detection image.
[0044] The present invention provides a computer device, which includes: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the steps of the method described in any one of the above-mentioned embodiments.
[0045] The present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.
[0046] The present invention provides a computer program product, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a computer device, the computer device is enabled to execute the steps of the method described in any one of the above embodiments.
[0047] An embodiment of the present specification provides a chip, including a storage unit and a processing unit, wherein the storage unit stores a computer program, and the processing unit implements the steps of the method described in any of the above embodiments when executing the computer program.
[0048] In the above-mentioned embodiment of the specification, first, the first feature point set of the image to be detected is obtained to match the feature points with the second feature point set of the reference image, and the target feature points in the first feature point set that match the second feature point set are obtained. If the number of target feature points is less than the preset number threshold, the distribution of the second feature points in the second feature point set on the reference image is determined. Then, clustering and feature point pairing are performed according to the distribution, the first feature point set and the second feature point set, and a plurality of feature point matching pairs whose number is greater than the preset number threshold are obtained. Finally, the image to be detected is corrected based on the matching of multiple feature points to obtain a corrected detection image. When the number of target feature points is less than the preset number threshold, it indicates that the matching result is not sufficient to accurately describe the relationship between the image to be detected and the reference image. Therefore, the distribution of the second feature points in the second feature point set on the reference image is determined, and clustering and feature point pairing are performed according to the distribution, the first feature point set and the second feature point set, and the spatial distribution of the feature points is comprehensively considered and the corresponding relationship between the image to be detected and the reference image is established, so as to obtain a plurality of feature point matches whose number is greater than the preset number threshold. Finally, the image to be detected is corrected based on the matching of multiple feature points whose number is greater than a preset threshold, which enhances the stability and reliability of the correction result, improves the quality of the corrected detection image, and provides more accurate image input for subsequent detection and recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1a A schematic diagram of a printed circuit board in a standard position provided for an embodiment of this specification;
[0050] Figure 1b A schematic diagram of a printed circuit board provided in an embodiment of the present specification in an up-and-down offset position;
[0051] Figure 1c A schematic diagram of a printed circuit board in a rotationally offset position provided in an embodiment of the present specification;
[0052] Figure 1d A flowchart of an image correction method provided in an embodiment of this specification;
[0053] Figure 1e A schematic diagram of a first characteristic point and a second characteristic point provided in an embodiment of this specification;
[0054] Figure 1f A schematic diagram of a corrected detection image provided in an embodiment of this specification;
[0055] Figure 2 A schematic diagram of a flow chart of an image correction method provided in an embodiment of this specification;
[0056] Figure 3 A schematic diagram of a process for obtaining a plurality of feature point matching pairs provided in an embodiment of this specification;
[0057] Figure 4 A schematic diagram of a process for obtaining a corrected detection image provided in an embodiment of this specification;
[0058] Figure 5a A schematic diagram of a process for obtaining a first feature point set provided in an embodiment of this specification;
[0059] Figure 5b A schematic diagram of the structure of a pre-trained object detection model provided in an embodiment of this specification;
[0060] Figure 6 A schematic diagram of a process for determining target feature points provided in an embodiment of this specification;
[0061] Figure 7 A schematic diagram of a process for determining whether to delete a feature point to be removed from a set of corrected feature points provided in an embodiment of this specification;
[0062] Figure 8 A schematic diagram of a flow chart of an image correction method provided in an embodiment of this specification;
[0063] Fig. 9 A schematic diagram of an image correction device provided in an embodiment of this specification;
[0064] Fig.10 An internal structural diagram of a computer device provided for an embodiment of this specification. DETAILED DESCRIPTION
[0065] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0066] In the field of industrial inspection, the application of computer vision technology is becoming more and more common. Among them, one of the common application scenarios is to test the qualification of products. The product is photographed by the shooting device on the inspection equipment, and the electronic image obtained is transmitted to the inspection software for processing to test the qualification of the product. However, the function of the shooting device on some inspection equipment is relatively lagging, resulting in blurred electronic images. In addition, the angle of the shooting device on the inspection equipment or the placement of the product causes the perspective tilt or deformation of the electronic image taken. These situations bring difficulties to subsequent inspection and identification. Therefore, in view of the above situation, it is necessary to perform image correction on the electronic image taken to ensure the accuracy and reliability of subsequent inspections.
[0067] In the related art, the four corner points corresponding to the electronic image are determined based on a deep convolutional neural network, and image correction is achieved based on the determined corner points. However, due to certain differences between the training data set and the real data set, the method based on the deep convolutional neural network has a poor effect in locating the four corner points in actual scenes. In addition, the method based on the deep convolutional neural network has high requirements for the amount of data, and will no longer be applicable when the amount of data is insufficient. In addition, it is also difficult for the method based on the deep convolutional neural network to achieve universal correction of perspective tilted images in various situations.
[0068] In the related art, the reference line of the image is determined, and the tilt angle of the reference line is calculated to find the average value. The average value is used as the tilt angle of the image. The image is then corrected using the tilt angle. However, the method of image correction based on the tilt angle is not suitable for the case where the image undergoes "perspective deformation". In addition, the method requires a threshold to be set in advance, has poor generalization and runs slowly.
[0069] In the related art, the influence of image noise on the direct Hough transform projection is firstly eliminated by accumulating the histogram of the image gradient direction, so as to obtain the rough tilt angle of the image. Then, the idea of Hough transform is used to project the image within the rough angle range, and then the precise image tilt angle is obtained, thereby reducing the angle projection range of the Hough transform. The above method mainly improves the problem of slow running speed of Hough transform by reducing the angle traversal interval of Hough transform. However, the image tilt correction method based on Hough transform and graded tilt detection optimization is suitable for correcting the situation of "rotational deformation" and depends on a pre-set threshold.
[0070] Based on the above analysis, the embodiment of the present specification provides an image correction method. First, a first feature point set of the image to be detected is obtained and a second feature point set of the reference image is matched with feature points to obtain target feature points in the first feature point set that match the second feature point set. If the number of target feature points is less than a preset number threshold, the distribution of the second feature points in the second feature point set on the reference image is determined. Then, clustering and feature point pairing are performed according to the distribution, the first feature point set and the second feature point set to obtain a plurality of feature point matching pairs whose number is greater than the preset number threshold. Finally, the image to be detected is corrected based on the matching of multiple feature points to obtain a corrected detection image. When the number of target feature points is less than the preset number threshold, it indicates that the matching result is not sufficient to accurately describe the relationship between the image to be detected and the reference image. Therefore, the distribution of the second feature points in the second feature point set on the reference image is determined and clustering and feature point pairing are performed according to the distribution, the first feature point set and the second feature point set, the spatial distribution of the feature points is comprehensively considered and the corresponding relationship between the image to be detected and the reference image is established to obtain a plurality of feature point matches whose number is greater than the preset number threshold. Finally, the image to be detected is corrected based on the matching of multiple feature points whose number is greater than a preset threshold, which enhances the stability and reliability of the correction result, improves the quality of the corrected detection image, and provides more accurate image input for subsequent detection and recognition tasks.
[0071] The image correction method provided in the embodiments of this specification can be applied to printed circuit board detection and recognition. The method can handle the situation where the printed circuit board is in an up-down offset position, a rotation offset position, and a 3D inspection. Figure 1a , Figure 1a The rectangular box 102 in FIG. 1 shows an image of a printed circuit board in a standard position. Figure 1b , Figure 1b The rectangular frame 104 in the figure shows an image of the printed circuit board in an up-down offset position, and the rectangular frame 102 shows an image of the printed circuit board in a standard position. Figure 1c , Figure 1c The rectangular frame 106 in the figure shows an image of the printed circuit board in a rotationally offset position, and the rectangular frame 102 shows an image of the printed circuit board in a standard position. It should be noted that the offset may be caused by the aging of the robot arm or the backward function resulting in inaccurate placement. The offset may cause deviations in the subsequent detection stage, so it is necessary to perform image correction on the image with the offset.
[0072] Specifically, see Figure 1d, use a camera to shoot the printed circuit board A to obtain the image to be detected. Perform target detection on the holes in the image to be detected to obtain the position data of multiple detection frames. Determine the center point of each detection frame according to the position data of each detection frame and form a first feature point set. The printed circuit board A corresponds to a reference image, and the reference image corresponds to a second feature point set based on the position data of the holes. Please refer to Figure 1e , Figure 1e The red feature point in is the second feature point of the reference image. Figure 1e The yellow feature point in the image is the first feature point of the image to be detected. Figure 1d , the first feature point set is corrected according to the coordinate deviation between the first feature point in the first feature point set and the second feature point in the second feature point set to obtain a corrected feature point set. Next, based on the distance data between the third feature point and the second feature point in the corrected feature point set, the feature points to be removed that meet the preset distance screening condition are determined in the corrected feature point set. Then, according to the corrected feature point set and the second feature point set, it is determined whether to delete the feature points to be removed from the corrected feature point set to determine the target feature points to be retained in the first feature point set, so as to achieve the pairing of feature points. Compare the size relationship between the number of target feature points and the preset number threshold 4, determine that the number of target feature points is less than the preset number threshold 4, then it is necessary to determine the distribution of the second feature points in the second feature point set on the reference image. Then, according to the distribution, the number of clusters is determined to be 12. Using the determined number of clusters 12, the second feature points existing in each cluster are determined on the reference image and used as the initial matching point of the cluster. The first feature point set is clustered using the number of clusters 12 to obtain 12 clusters. For any cluster, when the first feature point does not exist in the cluster, no feature point matching pair is generated. When the first feature point exists in the cluster, the first feature point in the cluster is matched with the initial matching point corresponding to the cluster to determine whether there is an initial matching point matching the first feature point in the cluster. When there is an initial matching point matching the first feature point, a feature point matching pair is generated. Traverse each cluster to determine the feature point matching pair corresponding to each cluster to achieve feature point pairing. Next, perform affine transformation based on the obtained multiple feature point matching pairs to obtain an affine transformation matrix. Finally, use the affine transformation matrix to correct the image to be detected to obtain a corrected detection image. Please refer to Figure 1f , Figure 1f Schematic diagram of the detected image after correction.
[0073] This specification provides an image correction method. Figure 2 , the method may include the following steps:
[0074] S210, acquiring a first feature point set of the image to be detected and performing feature point matching with a second feature point set of the reference image to obtain target feature points in the first feature point set that match the second feature point set.
[0075] The image to be detected corresponds to a reference image. The image to be detected may be an image containing an object to be detected. The reference image may be a standard image taken of the object to be detected.
[0076] Specifically, a pre-trained target detection model is used to perform target detection on the image to be detected, and the target object is identified in the image to be detected, so as to determine the image detection frame corresponding to the target object in the image to be detected. Next, for each image detection frame, the first feature point of each image detection frame is determined. The first feature point set is formed by the first feature point of each image detection frame. Among them, the pre-trained target detection model can be a neural network model that meets the real-time detection speed and the production speed of the production line. For example, the preset trained target detection model can be constructed and trained based on YOLO, FasterR-CNN, etc.
[0077] In order to more accurately identify the target object in the image to be detected, a reference image is required to improve the accuracy. The reference image corresponding to the image to be detected is a standard image taken of the object to be detected. The target object is marked by using a rectangular frame in the reference image. For each rectangular frame, its center point is calculated. The center point of each rectangular frame is used as the second feature point to form a second feature point set. The first feature point set of the image to be detected is matched with the second feature point set of the reference image using a pre-set matching rule to obtain a feature point matching result. The feature point matching result is evaluated to determine the feature point matching result that meets the requirements, and the first feature point corresponding to the feature point matching result that meets the requirements is used as the target feature point that matches a second feature point in the second feature point set. Among them, the pre-set matching rules can be set according to actual conditions and needs.
[0078] S220: If the number of target feature points is less than a preset number threshold, determine the distribution of second feature points in the second feature point set on the reference image.
[0079] The preset quantity threshold may be a set value used to determine whether the matching result meets the requirements during the feature point matching process. The distribution condition may be a spatial distribution feature of the second feature point set in the reference image.
[0080] Specifically, a preset number threshold is set according to actual conditions and requirements. If the number of target feature points is less than the preset number threshold, it means that the matching effect is not ideal because the matched feature points are too few and cannot provide sufficient information for subsequent processing. Therefore, feature point matching needs to be performed again. The feature point matching process is assisted by analyzing the distribution of the second feature points in the second feature point set on the reference image, in order to obtain more effective target feature points.
[0081] S230 , clustering and pairing feature points according to the distribution, the first feature point set, and the second feature point set to obtain a plurality of feature point matching pairs.
[0082] S240 , correcting the image to be detected based on matching of multiple feature points to obtain a corrected detection image.
[0083] The number of feature point matching pairs is greater than a preset number threshold.
[0084] Specifically, the first feature point set is clustered according to the distribution, and is divided into a plurality of clusters to obtain a clustering result. Then, the similarity or distance between the first feature point in the cluster and the second feature point in the second feature point set is compared to determine the matching relationship between the feature points, thereby obtaining a plurality of feature point matching pairs. Then, based on the plurality of feature point matching pairs, a suitable transformation model (such as affine transformation, perspective transformation) is selected. Next, the parameters of the transformation model are determined by using the plurality of feature point matching pairs and the selected transformation model for calculation. By traversing each pixel position in the image to be detected, and then using the parameters of the transformation model and the coordinate transformation formula, the new position corresponding to each pixel position is calculated to obtain the corrected detection image.
[0085] In the above-mentioned implementation, first, the first feature point set of the image to be detected is obtained to match the feature points with the second feature point set of the reference image, and the target feature points in the first feature point set that match the second feature point set are obtained. If the number of target feature points is less than the preset number threshold, the distribution of the second feature points in the second feature point set on the reference image is determined. Then, clustering and feature point pairing are performed according to the distribution, the first feature point set and the second feature point set, and a plurality of feature point matching pairs whose number is greater than the preset number threshold are obtained. Finally, the image to be detected is corrected based on the matching of multiple feature points to obtain a corrected detection image. When the number of target feature points is less than the preset number threshold, it indicates that the matching result is not sufficient to accurately describe the relationship between the image to be detected and the reference image. Therefore, the distribution of the second feature points in the second feature point set on the reference image is determined, and clustering and feature point pairing are performed according to the distribution, the first feature point set and the second feature point set, and the spatial distribution of the feature points is comprehensively considered and the corresponding relationship between the image to be detected and the reference image is established, so as to obtain a plurality of feature point matches whose number is greater than the preset number threshold. Finally, the image to be detected is corrected based on the matching of multiple feature points whose number is greater than a preset threshold, which enhances the stability and reliability of the correction result, improves the quality of the corrected detection image, and provides more accurate image input for subsequent detection and recognition tasks.
[0086] In some embodiments, see Figure 3 , clustering and pairing feature points according to the distribution, the first feature point set, and the second feature point set to obtain a plurality of feature point matching pairs may include the following steps:
[0087] S310: Determine the number of clusters according to the distribution situation.
[0088] S320 , clustering the first feature point set according to the number of clusters to obtain a plurality of clusters.
[0089] S330 , pairing feature points using each cluster and feature points in the second feature point set to obtain a plurality of feature point matching pairs.
[0090] Among them, clustering can refer to dividing the data points in the data set into different groups according to a certain similarity criterion, so that the similarity between the data points in the group is high, while the similarity between the groups is low.
[0091] Specifically, according to the distribution, a suitable method (such as elbow rule, silhouette coefficient, gapstatistic) is selected to divide the second feature point set to form different sets, each set is a cluster, so as to determine the number of clusters. The second feature point set can also be directly divided according to the observed distribution to determine the number of clusters. After determining the number of clusters, it is necessary to select a suitable clustering algorithm (such as K-means clustering, DBSCAN), use the determined number of clusters, cluster the first feature point set, and obtain multiple clusters. For any cluster among the multiple clusters, the cluster is matched with the feature points in the second feature point set using the matching rules to obtain the matching result corresponding to the cluster. The matching result is analyzed to determine whether the cluster has a second feature point that matches it. If the matching result indicates that there is a second feature point that matches the cluster, a feature point matching pair is obtained. Traverse each cluster, repeat the above operation, determine whether each cluster has a second feature point that matches it, so as to obtain a feature point matching pair corresponding to each cluster. Since there are multiple clusters, multiple feature point matching pairs can be obtained.
[0092] Exemplarily, the number of clusters is determined to be K according to the distribution. K coordinate points are randomly selected as the starting cluster center points. For any first feature point in the first feature point set, the distance data between the first feature point and the K starting cluster center points is calculated. The distance data between the first feature point and the K starting cluster center points are compared, the minimum distance data is determined, and the first feature point is assigned to the cluster corresponding to the starting cluster center point corresponding to the minimum distance data. Each first feature point in the first feature point set is traversed, and the above operation is repeated to assign each feature point to its corresponding cluster. Then, according to the first feature point in each cluster cluster, the mean of all its first feature points is calculated and used as the center point of the cluster cluster. Then, the center point of each cluster cluster is used as the new cluster center point, and the first feature point set is re-clustered. Repeat the above operation until the maximum number of iterations is reached or the cluster center point no longer changes significantly, and clustering is stopped to obtain multiple clusters.
[0093] In the above implementation, the number of clusters is determined according to the distribution, and the first feature point set is clustered according to the number of clusters to obtain multiple clusters. Each cluster is paired with the feature points in the second feature point set to obtain multiple feature point matching pairs, which provide a data basis for the subsequent determination of the affine transformation matrix and the realization of image correction.
[0094] In some embodiments, each cluster has an initial matching point corresponding to it in the second feature point set. Using each cluster to pair feature points with feature points in the second feature point set to obtain multiple feature point matching pairs may include: matching each cluster with the initial matching point corresponding to each cluster to obtain multiple feature point matching pairs.
[0095] The initial matching points may be second feature points included in each cluster when the second feature point set is divided into a plurality of clusters according to the distribution of the second feature points before the feature point matching is performed.
[0096] Specifically, each clustering cluster is matched with the initial matching point corresponding to the clustering cluster using a pre-set matching rule to obtain a matching result corresponding to the clustering cluster. The matching result is analyzed to determine whether there is an initial matching point that matches the clustering cluster. If the matching result indicates that there is an initial matching point that matches the clustering cluster, a feature point matching pair is obtained. Each clustering cluster is traversed, and the above operation is repeated to determine whether each clustering cluster has an initial matching point that matches it, so as to obtain a feature point matching pair corresponding to each clustering cluster. Since there are multiple clustering clusters, multiple feature point matching pairs can be obtained. Among them, the pre-set matching rule can be set according to the actual situation and needs. For example, the pre-set matching rule can be the Euclidean distance between the clustering cluster and the initial matching point.
[0097] In the above implementation, each cluster is matched with the initial matching point corresponding to each cluster to obtain a plurality of feature point matching pairs, which provide a data basis for the subsequent determination of the affine transformation matrix and the realization of image correction.
[0098] In some embodiments, the first feature point set includes the first feature point obtained by performing target detection on the image to be detected. Matching each cluster with the initial matching point corresponding to each cluster to obtain multiple feature point matching pairs may include: if any cluster is not empty, matching based on the distance data between the first feature point in any cluster and the initial matching point to obtain feature point matching pairs of at least part of the first feature points in any cluster.
[0099] Specifically, if there is at least one first feature point in any cluster, the cluster is considered not to be empty. Next, a distance measurement method (such as cosine similarity, dot product, Euclidean distance, etc.) is used to calculate the distance data between the first feature point in the cluster and the initial matching point corresponding to the cluster, and the minimum distance data is determined. Therefore, the initial matching point corresponding to the minimum distance data and the first feature point form a feature point matching pair. Since the number of first feature points in the cluster may be inconsistent with the number of initial matching points corresponding to the cluster, not every first feature point has an initial matching point that matches it, so at least some of the first feature points in the cluster can determine the initial matching points that match them and form a feature point matching pair.
[0100] In the above implementation, if any cluster is not empty, matching is performed based on the distance data between the first feature point in any cluster and the initial matching point to obtain a feature point matching pair of at least part of the first feature points in any cluster, thereby providing a data basis for the subsequent determination of the affine transformation matrix and realizing image correction.
[0101] In some embodiments, the first feature point set includes first feature points obtained by performing target detection on the image to be detected. Each cluster is matched with the initial matching point corresponding to each cluster to obtain multiple feature point matching pairs, which may include: if any cluster is empty, no feature point matching pair is generated.
[0102] Specifically, if there is no first feature point in any cluster, that is, the cluster is empty, it can be considered that there is no first feature point matching the second feature point in the reference image in the image to be detected, so the cluster does not generate a feature point matching pair.
[0103] In the above implementation, if any cluster is empty, no feature point matching pair is generated, which provides a data basis for the subsequent determination of the affine transformation matrix and the realization of image correction.
[0104] In some embodiments, the first feature point set includes first feature points obtained by performing target detection on the image to be detected. Matching each cluster with the initial matching point corresponding to each cluster to obtain multiple feature point matching pairs may include: if there is a first feature point in any cluster, then determining a matching feature point that matches a first feature point in the initial matching points corresponding to any cluster, and correspondingly forming a feature point matching pair.
[0105] Specifically, if there is a first feature point in any cluster, and the cluster corresponds to an initial matching point, the initial matching point can be directly used as the matching feature point that matches the first feature point. If there is a first feature point in any cluster, but the cluster corresponds to multiple initial matching points, the distance measurement method (such as cosine similarity, dot product, Euclidean distance, etc.) is used to calculate the distance data between the first feature point and each initial matching point. Compare the distance data between the first feature point and each initial matching point, determine the minimum distance data, and use the initial matching point corresponding to the minimum distance data as the matching feature point that matches the first feature point. Therefore, the first feature point and the matching feature point constitute a feature point matching pair.
[0106] In the above implementation, if there is a first feature point in any cluster, a matching feature point that matches the first feature point is determined in the initial matching points corresponding to any cluster, and a corresponding feature point matching pair is formed to provide a data basis for the subsequent determination of the affine transformation matrix and the realization of image correction.
[0107] In some embodiments, the first feature point set includes first feature points obtained by performing target detection on the image to be detected. Matching each cluster with the initial matching point corresponding to each cluster to obtain multiple feature point matching pairs may include: if there are multiple first feature points in any cluster, and any cluster corresponds to an initial matching point, determining a first feature point that matches an initial matching point among the multiple first feature points, and correspondingly forming a feature point matching pair.
[0108] Specifically, if there are multiple first feature points in any cluster, and the cluster corresponds to an initial matching point, the distance measurement method (such as cosine similarity, dot product, Euclidean distance, etc.) is used to calculate the distance data between each first feature point and the initial matching point. The distance data between each first feature point and the initial matching point is compared to determine the minimum distance data, and the first feature point corresponding to the minimum distance data is used as the first feature point matching the initial matching point. Therefore, the first feature point matching the initial matching point and the initial matching point constitute a feature point matching pair.
[0109] In the above implementation, if there are multiple first feature points in any cluster, and any cluster corresponds to an initial matching point, a first feature point matching with an initial matching point is determined among the multiple first feature points, and a corresponding feature point matching pair is formed, which provides a data basis for the subsequent determination of the affine transformation matrix and the realization of image correction.
[0110] In some embodiments, the first feature point set includes first feature points obtained by performing target detection on the image to be detected. Matching each cluster with the initial matching point corresponding to each cluster to obtain multiple feature point matching pairs may include: if there are multiple first feature points in any cluster, and any cluster corresponds to multiple initial matching points, for any first feature point in any cluster, determining the matching feature points that match any first feature point in the initial matching points corresponding to any cluster, and correspondingly constituting the feature point matching pairs of any first feature point.
[0111] Specifically, if there are multiple first feature points in any cluster, and the cluster corresponds to multiple initial matching points, for any first feature point among the multiple first feature points in any cluster, the distance measurement method (such as cosine similarity, dot product, Euclidean distance, etc.) is used to calculate the distance data between the first feature point and each initial matching point. The distance data between the first feature point and each initial matching point is compared to determine the minimum distance data S1, and the initial matching point corresponding to the minimum distance data S1 is used as the matching feature point matching the first feature point. The first feature point and the matching feature point constitute a feature point matching pair. Then the distance data between another first feature point and each initial matching point is calculated, and the minimum distance data S2 is determined. If the initial matching point corresponding to the minimum distance data S2 is the same as the initial matching point corresponding to the minimum distance data S1, the minimum distance data S2 is compared with the minimum distance data S1, and the first feature point corresponding to the minimum value is used as the first feature point matching the initial matching point and constitutes a feature point matching pair with the initial matching point. If the initial matching point corresponding to the minimum distance data S2 is not the same as the initial matching point corresponding to the minimum distance data S1, the initial matching point corresponding to the minimum distance data S2 is used as the matching feature point that matches the other first feature point. The other first feature point and the matching feature point constitute a feature point matching pair. Traverse each first feature point in any cluster, repeat the above operation, determine whether there is a matching feature point for any first feature point, and form a feature point matching pair corresponding to it.
[0112] In the above implementation, if there are multiple first feature points in any clustering cluster, and any clustering cluster corresponds to multiple initial matching points, for any first feature point in any clustering cluster, the matching feature points that match any first feature point are determined among the initial matching points corresponding to any clustering cluster, and the corresponding feature point matching pairs that constitute any first feature point are determined, so as to provide a data basis for the subsequent determination of the affine transformation matrix and the realization of image correction.
[0113] In some embodiments, see Figure 4 , correcting the image to be detected based on the matching of multiple feature points to obtain a corrected detection image, which may include the following steps:
[0114] S410 , performing affine transformation based on multiple feature point matching pairs to obtain an affine transformation matrix.
[0115] Among them, the affine transformation can be a linear transformation that maintains the parallelism and collinearity of the original objects in the image. Under the affine transformation, straight lines remain straight lines, parallel lines remain parallel, and objects in the original image maintain their shapes. The affine transformation matrix can be a matrix that describes the transformation rules of an image under the affine transformation, and is used to represent geometric transformations such as translation, rotation, scaling, and shearing of an image.
[0116] Specifically, after determining a plurality of feature point matching pairs whose number is greater than the preset number threshold, the corresponding relationship between the plurality of feature point matching pairs and the transformation matrix are used for calculation to obtain an affine transformation matrix.
[0117] For example, under affine transformation, the translation vector can be set to t, and each point can be moved to (x+t, y+t). The corresponding transformation matrix under the translation condition is:
[0118] Under affine transformation, the origin of the rotation transformation can be set to rotate the target figure clockwise around the origin by θ radians. The corresponding transformation matrix under the rotation condition is:
[0119] By combining the translation and rotation transformations through matrix multiplication, we can obtain the affine transformation matrix M of multiple feature point matching pairs after rotation and translation. The affine transformation matrix M is expressed as: After calculation, the affine transformation matrix is obtained In order to facilitate the derivation,
[0120] According to the coordinate data of multiple feature point matching pairs, the coordinates of the first feature point in the feature point matching pair are set to (x, y), and the coordinates of the second feature point in the feature point matching pair are set to (x, y). The formula can be obtained:
[0121] Taking n feature point matching pairs as an example, let the horizontal coordinates of the first feature point in the n feature point matching pairs be x1, x2, …x n That is, x=(x1,x2,…x n ), the ordinates are y1, y2, …y n That is, y=(y1,y2,…y n ), the horizontal coordinates of the second feature point in the n feature point matching pairs are Right now The vertical axes are Right now The affine transformation matrix x=(x1,x2,…x n )、y=(y1,y2,…y n ), Substitution The following system of equations is obtained:
[0122]
[0123]
[0124]
[0125]
[0126] Solve the above equations to determine a 11 ,a 12 ,a 13 ,a 21 ,a 22 ,a 23 The corresponding values are obtained to determine the affine transformation matrix M.
[0127] S420, correcting the image to be detected by using an affine transformation matrix to obtain a corrected detection image.
[0128] Specifically, each pixel position in the image to be detected is corrected using an affine transformation matrix to obtain a corrected pixel position. Then, according to an interpolation method (such as bilinear interpolation), the pixel value corresponding to the corrected pixel position is calculated based on the pixel value of each pixel position in the image to be detected to obtain a corrected detection image.
[0129] Exemplarily, the coordinates of each pixel position in the image to be detected are expressed as (m, n). Correct the coordinates of each pixel position in the image to be detected to obtain the corrected coordinates corresponding to each pixel position Thus, the corrected detection image is obtained.
[0130] In the above implementation, an affine transformation is performed based on multiple feature point matching pairs to obtain an affine transformation matrix, and the image to be detected is corrected using the affine transformation matrix to obtain a corrected detection image, thereby improving the quality of the corrected detection image and improving the accuracy of subsequent detection and recognition.
[0131] In some embodiments, see Figure 5a , the first feature point set is obtained by:
[0132] S510 , performing target detection on the image to be detected, and obtaining position data of multiple detection frames.
[0133] S520 . Determine a center point of each detection frame based on the position data of each detection frame.
[0134] S530: Construct a first feature point set based on the center point of each detection frame.
[0135] The detection box may be a rectangular bounding box generated by an object detection algorithm (such as an object detector) in the image to be detected, which is used to identify the area in the image to be detected that may contain the target object. The position data of the detection box may be information describing the position and size of the detection box in the image to be detected.
[0136] Specifically, an image acquisition device is used to shoot an object to be detected to obtain an image to be detected. Then, a pre-trained target detection model is used to perform target detection on the image to be detected, and the target object is identified in the image to be detected, so as to determine the detection frame corresponding to the target object in the image to be detected and the position data corresponding to the detection frame, such as the image detection frame can be the minimum circumscribed rectangle corresponding to the target object. In general, there are multiple target objects in the image to be detected, so a pre-trained target detection model is used to perform target detection on the image to be detected, and multiple detection frames and the position data corresponding to each detection frame are obtained. For each detection frame, the center point of the detection frame is calculated according to its corresponding position data. The center point of each detection frame is used as the first feature point to form a first feature point set. Among them, the image acquisition device can be any one of a camera, a video camera, a fisheye camera, and a surveillance camera. The pre-trained target detection model can be a neural network model that meets the real-time detection speed and the production speed of the production line assembly line. For example, the preset trained target detection model can be constructed and trained based on YOLO, Faster R-CNN, etc.
[0137] Exemplarily, the object to be detected may be a printed circuit board (PCB). On the printed circuit board, the target object may be a hole on the printed circuit board. Generally, the holes on the printed circuit board are located at the four corners of the board and some specific middle positions. The printed circuit board is placed in a viewfinder, and the industrial camera in the viewfinder is fixed directly above the printed circuit board. The lighting in the viewfinder is multi-directional. The printed circuit board is photographed using the industrial camera in the viewfinder to obtain an image to be detected.
[0138] In some embodiments, the pre-trained object detection model can be a deep convolutional neural network trained based on yolov5. Figure 5b , Figure 5bThis is a schematic diagram of the network structure of a deep convolutional neural network trained based on yolov5. The network is mainly composed of a focus module, a CBL module (Conv-Bn-LeakyRelu, a convolutional block with batch normalization and leaky corrected linear units), a CSP module (Cross Stage Partial, cross-stage local network) including CSP1_x and CSP2_x, and an SPP module (Spatial Pyramid Pooling). Among them, the focus module mainly includes four parallel slice layers for processing input images. The CBL module includes a convolution layer, a batch normalization layer, and an activation function. The CSP1_x module includes a CBL module and x residual connection units. The CSP2_x module includes x CBL modules. The SPP module mainly includes three maxpool layers.
[0139] Before obtaining a deep convolutional neural network trained based on yolov5, a data set for training needs to be obtained. Specifically, annotators are required to annotate the original image containing the object to be detected (such as a printed circuit board). The annotators find the target object in the original image (such as a hole on a printed circuit board) and use a rectangular frame to select the target object along the outer edge of the target object. The original image with label information including the target object and the corresponding rectangular frame is divided into a training set and a test set in a ratio of 8:2. It should be noted that in order to improve the robustness and generalization ability of the model, the original image can be digitized (such as denoising, image enhancement, image filtering), and the original image can be flipped at random angles and mirrors. In addition, in order to simulate different shooting conditions and lighting problems, the original image can also be randomly edge filled, randomly inverted, and random noise added. It should be emphasized that the above data processing process can be performed in an offline environment.
[0140] After determining the data set for training, in order to accelerate model training and reasoning, it is necessary to control the image size in the data set within a reasonable range (for example, the image size is controlled within 1200*1000), which can reduce the amount of calculation when the model extracts image features, thereby improving the speed of training and reasoning. Therefore, the target detection model to be trained is constructed according to the demand standards, and then the constructed target detection model to be trained is trained using the data set. When the model training stop condition is reached, the trained target detection model is obtained. Exemplarily, the training parameters of the target detection model to be trained can be as follows: Lr (learning rate) = 0.0025, momentum (momentum) = 0.0009, weight_decay (weight decay) = 0.0001, optimizer (optimizer) = SGD (stochastic gradient descent), epoch (number of iterations) = 60, batchsize (batch size) = 4.
[0141] For the trained target detection model, the performance of the model is evaluated using the divided 20% test set. The evaluation indicators may include box accuracy (such as Intersection over Union (IoU)). Box accuracy can be an indicator of the model's ability to accurately locate the target object. After the evaluation is completed, the evaluation indicator data corresponding to the target detection model will be labeled. The target detection model corresponding to the highest evaluation indicator data is selected from multiple trained target detection models as the pre-trained target detection model in practical applications.
[0142] In the above implementation, target detection is performed on the image to be detected to obtain position data of multiple detection frames, the center point of each detection frame is determined based on the position data of each detection frame, and a first feature point set is constructed based on the center point of each detection frame to provide a data basis for subsequent feature point matching.
[0143] In some embodiments, see Figure 6 , the target feature points are determined by:
[0144] S610 , correcting the first feature point set according to a coordinate deviation between a first feature point in the first feature point set and a second feature point in the second feature point set to obtain a corrected feature point set.
[0145] The corrected feature point set may be a new feature point list formed after correcting or adjusting the detected first feature point. The coordinate deviation may be a quantification of the difference between the detected feature point and the template feature point.
[0146] Specifically, for any first feature point in the first feature point set, the horizontal coordinate deviation between the horizontal coordinate of the first feature point and the horizontal coordinate of each second feature point in the second feature point set and the vertical coordinate deviation between the vertical coordinate of the first feature point and the vertical coordinate of each second feature point in the second feature point set are calculated. The horizontal coordinate deviation between the horizontal coordinate of the first feature point and each second feature point in the second feature point set is compared to determine the minimum horizontal coordinate deviation. The vertical coordinate deviation between the horizontal coordinate of the first feature point and each second feature point in the second feature point set is compared to determine the minimum vertical coordinate deviation. The minimum horizontal coordinate deviation and the minimum vertical coordinate deviation corresponding to the first feature point constitute the coordinate deviation. Then, the coordinate of the first feature point is corrected by using the coordinate deviation, and the corrected coordinate of the first feature point is obtained as the third feature point. Each first feature point in the first feature point set is traversed, and the coordinates are corrected by using the coordinate deviation corresponding to each first feature point to obtain the third feature point corresponding to each first feature point. The third feature points corresponding to each first feature point in the first feature point set constitute the corrected feature point set.
[0147] S620: Determine, in the corrected feature point set, feature points to be removed that meet a preset distance screening condition based on the distance data between the third feature point and the second feature point in the corrected feature point set.
[0148] The preset distance screening condition may be a condition for controlling the matching accuracy in the feature point matching process, for example, the preset distance screening condition may be a threshold. The feature points to be eliminated may be feature points that are considered to be likely to deviate greatly from the template feature points.
[0149] Specifically, a preset distance filtering condition is set according to actual conditions and requirements. For any third feature point in the corrected feature point set, calculate the distance data between the third feature point and each second feature point in the second feature point set. Compare the distance data between the third feature point and each second feature point in the second feature point set, determine the minimum distance data therein, and compare the minimum distance data with the preset distance filtering condition. If the comparison result indicates that the minimum distance data meets the preset distance filtering condition, the third feature point is used as a feature point to be eliminated. If the comparison result indicates that the minimum distance data does not meet the preset distance filtering condition, determine whether the next third feature point is a feature point to be eliminated. Traverse each third feature point in the corrected feature point set, repeat the above operation, and determine the feature points to be eliminated that meet the preset distance filtering condition in the corrected feature point set.
[0150] S630: Determine whether to delete the feature points to be removed from the revised feature point set according to the revised feature point set and the second feature point set, so as to determine the target feature points that need to be retained in the first feature point set.
[0151] Specifically, according to the pre-set screening rules, the feature points to be removed are used to screen the revised feature point set and the second feature point set to obtain the corresponding filtered sets. Then, based on the pre-set matching rules, the filtered sets corresponding to the revised feature point set and the second feature point set are matched, and the matching results are compared with the pre-set deletion criteria. According to the comparison results, it is determined whether to delete the feature points to be removed from the revised feature point set to determine the third feature point that needs to be retained in the revised feature point set. Then, the retained third feature point is used to determine the first feature point corresponding to it in the first feature point set as the target feature point.
[0152] It should be noted that the pre-set filtering rules can be used to select the correction feature points that meet the standards in the correction feature point list. The pre-set filtering rules can be selected and defined according to specific application scenarios and requirements. For example, the pre-set filtering rules can be to directly delete the feature points to be verified from the correction feature point list, or to retain the correction feature points greater than the threshold based on the distance threshold. The pre-set deletion criteria can be used to determine which matching results meet the requirements so that the feature points to be verified can be deleted from the correction feature point list. The pre-set deletion criteria can be selected and defined according to specific application scenarios and requirements. For example, the pre-set deletion criteria can filter the correction list and the template feature points that match each filtered feature point in the template feature point list to ensure that the geometric relationship is in line with expectations.
[0153] In the above-mentioned implementation manner, the first feature point set is corrected according to the coordinate deviation between the first feature point in the first feature point set and the second feature point in the second feature point set to obtain a corrected feature point set, and the feature points to be eliminated that meet the preset distance screening condition are determined in the corrected feature point set based on the distance data between the third feature point and the second feature point in the corrected feature point set, and it is determined whether to delete the feature points to be eliminated from the corrected feature point set according to the corrected feature point set and the second feature point set, so as to determine the target feature points to be retained in the first feature point set, and by correcting the first feature point set, the feature points to be eliminated that need to be further verified as to whether they can be used for feature point pairing are determined based on the distance data between the third feature point and the second feature point in the corrected feature point set, and then it is determined whether the feature points to be eliminated are used for feature point matching according to the feature points to be eliminated and the second feature point set, and further, the target feature points that match the first feature points are determined, thereby improving the accuracy of feature point matching.
[0154] In some embodiments, see Figure 7 , determining whether to delete the feature points to be removed from the revised feature point set according to the revised feature point set and the second feature point set may include the following steps:
[0155] S710 , attempting to delete the feature points to be removed from the corrected feature point set to obtain a deleted feature point set.
[0156] Specifically, since the feature point to be removed is selected from the modified feature point set, there is a third feature point in the modified feature point set that is consistent with the feature point to be removed. The modified feature point set is traversed, and the third feature point that is consistent with the feature point to be removed is determined in the modified feature point set, and the third feature point is deleted from the modified feature point set. After deleting the third feature point, the remaining third feature points in the modified feature point set constitute the deleted feature point set.
[0157] S720. Determine, in the second feature point set, verification feature points corresponding to each feature point in the deleted feature point set to obtain a verification feature point set.
[0158] Specifically, for any feature point in the deleted feature point set, calculate the distance data between the feature point and each second feature point in the second feature point set. Compare the distance data between the feature point and each second feature point in the second feature point set, determine the minimum distance data therein, and select the second feature point corresponding to the minimum distance data from the second feature point set as the verification feature point corresponding to the feature point. Traverse each feature point in the deleted feature point set, repeat the above operation, and determine the verification feature point corresponding to each feature point. The verification feature points corresponding to each feature point in the deleted feature point set constitute the verification feature point set.
[0159] S730: If the sum of the distance deviations between the feature point set after deletion and the verified feature point set is less than the deviation and threshold, determine to delete the feature point to be deleted.
[0160] S740: If the sum of the distance deviations between the feature point set after deletion and the verified feature point set is not less than the deviation sum threshold, determine not to delete the feature point to be removed.
[0161] Among them, the distance deviation and the threshold can be used to evaluate the accuracy of the matching of the feature points to be removed. The deviation and the threshold can be used as a standard or threshold to evaluate the matching quality of the feature points to be removed.
[0162] Specifically, for any third feature point in the deleted feature point set, the distance data between the third feature point and each verification feature point in the verification feature point set is calculated. The distance data between the third feature point and each verification feature point in the verification feature point set is compared to determine the minimum distance data therein, and the two feature points corresponding to the minimum distance data are used as a matching feature point. Each third feature point in the deleted feature point set is traversed, and the above operation is repeated to determine multiple matching feature points. Each matching feature point pair consists of two feature points. Each matching feature point pair is traversed, and the distance data between the two feature points in each matching feature point pair is calculated. By adding the distance data of each matching feature point pair, the sum operation of the distance data is realized to obtain the distance deviation sum. A deviation sum threshold is pre-set to measure whether the feature point to be removed needs to be deleted. The size of the distance deviation sum is compared with the deviation sum threshold. If the distance deviation sum is less than the deviation sum threshold, it is considered that the distance between the feature point to be removed deviates from the standard feature point, that is, the second feature point, is large, which affects the accuracy of the feature point matching, so it is necessary to delete the feature point to be removed from the corrected feature point set. If the distance deviation is not less than the deviation and threshold, it is considered that the distance between the feature point to be removed and the standard feature point, i.e., the second feature point, is relatively close and does not affect the accuracy of feature point matching. Therefore, there is no need to delete the feature point to be removed from the corrected feature point set.
[0163] In the above implementation, an attempt is made to delete the feature points to be eliminated from the corrected feature point set to obtain a deleted feature point set, and verification feature points corresponding to each feature point in the deleted feature point set are determined in the second feature point set to obtain a verified feature point set. If the sum of the distance deviations between the deleted feature point set and the verified feature point set is less than the deviation and the threshold, it is determined that the feature points to be eliminated are deleted; if the sum of the distance deviations between the deleted feature point set and the verified feature point set is not less than the deviation and the threshold, it is determined that the feature points to be eliminated are not deleted, thereby improving the accuracy of feature point matching.
[0164] This specification also provides an image correction method. For example, see Figure 8 , the image correction method may include the following steps:
[0165] S802: Perform object detection on the image to be detected to obtain position data of multiple detection frames.
[0166] S804: Determine the center point of each detection frame based on the position data of each detection frame.
[0167] S806: Construct a first feature point set based on the center point of each detection frame.
[0168] The first feature point set includes first feature points obtained by performing target detection on the image to be detected.
[0169] S808 . Correct the first feature point set according to a coordinate deviation between a first feature point in the first feature point set and a second feature point in the second feature point set of the reference image to obtain a corrected feature point set.
[0170] S810: Determine, in the corrected feature point set, feature points to be removed that meet a preset distance screening condition based on the distance data between the third feature point and the second feature point in the corrected feature point set.
[0171] S812: Try to delete the feature points to be removed from the corrected feature point set to obtain a deleted feature point set.
[0172] S814. Determine in the second feature point set the verification feature points corresponding to each feature point in the deleted feature point set to obtain a verification feature point set.
[0173] S816. Determine whether to delete the feature points to be removed from the corrected feature point set based on the distance deviation and the deviation and threshold between the deleted feature point set and the verified feature point set, so as to determine the target feature points that need to be retained in the first feature point set.
[0174] S818: If the number of target feature points is less than a preset number threshold, determine the distribution of second feature points in the second feature point set on the reference image.
[0175] S820: Determine the number of clusters according to the distribution situation.
[0176] S822: Cluster the first feature point set according to the number of clusters to obtain a plurality of clusters.
[0177] Each cluster has a corresponding initial matching point in the second feature point set.
[0178] S824. If any cluster is empty, no feature point matching pair is generated; if any cluster is not empty, matching is performed based on the distance data between the first feature point in any cluster and the initial matching point to obtain a feature point matching pair of at least part of the first feature point in any cluster.
[0179] The number of feature point matching pairs is greater than a preset number threshold.
[0180] S826. Perform affine transformation based on multiple feature point matching pairs to obtain an affine transformation matrix.
[0181] S828. Correct the image to be detected using an affine transformation matrix to obtain a corrected detection image.
[0182] This specification provides an image correction device 900. Fig. 9The image correction device 900 includes: a feature point matching module 910, a distribution determination module 920, a feature point pairing module 930, and a to-be-detected image correction module 940.
[0183] A feature point matching module 910 is used to obtain a first feature point set of the image to be detected and perform feature point matching with a second feature point set of the reference image to obtain target feature points in the first feature point set that match the second feature point set;
[0184] A distribution determination module 920, configured to determine the distribution of the second feature points in the second feature point set on the reference image if the number of the target feature points is less than a preset number threshold;
[0185] A feature point pairing module 930 is used to perform clustering and feature point pairing according to the distribution, the first feature point set and the second feature point set to obtain a plurality of feature point matching pairs; wherein the number of the feature point matching pairs is greater than the preset number threshold;
[0186] The image to be detected correction module 940 is used to correct the image to be detected based on the matching of the multiple feature points to obtain a corrected detection image.
[0187] For a detailed description of the image correction device, please refer to the description of the image correction method above, which will not be repeated here.
[0188] An embodiment of the present specification provides a chip including a storage unit and a processing unit. The storage unit stores a computer program. When the processing unit executes the computer program, the steps of the method of any of the above embodiments are implemented.
[0189] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image correction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0190] Those skilled in the art will understand that Fig.10 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution disclosed in this specification, and does not constitute a limitation on the computer device to which the solution disclosed in this specification is applied. Specifically, the computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0191] An embodiment of the present specification provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method steps in the above embodiment are implemented.
[0192] An embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.
[0193] The embodiments of this specification provide a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of any method of the above embodiments.
[0194] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
Claims
1. An image correction method, characterized in that: The method comprises: Acquire a first feature point set of the image to be detected and perform feature point matching with a second feature point set of the reference image to obtain target feature points in the first feature point set that match the second feature point set; If the number of the target feature points is less than a preset number threshold, determining the distribution of the second feature points in the second feature point set on the reference image; Clustering and feature point pairing are performed according to the distribution, the first feature point set, and the second feature point set to obtain a plurality of feature point matching pairs; wherein the number of the feature point matching pairs is greater than the preset number threshold; The image to be detected is corrected based on the matching of the multiple feature points to obtain a corrected detection image.
2. The method according to claim 1, characterized in that The clustering and feature point pairing according to the distribution, the first feature point set, and the second feature point set to obtain a plurality of feature point matching pairs includes: Determine the number of clusters according to the distribution; Clustering the first feature point set according to the number of clusters to obtain a plurality of clusters; Each cluster is used to pair feature points with feature points in the second feature point set to obtain the plurality of feature point matching pairs.
3. The method according to claim 2, characterized in that Each of the clusters corresponds to an initial matching point in the second feature point set; The step of pairing feature points using each cluster with the feature points in the second feature point set to obtain the plurality of feature point matching pairs includes: Each cluster is matched with an initial matching point corresponding to each cluster to obtain the plurality of feature point matching pairs.
4. The method according to claim 3, characterized in that The first feature point set includes first feature points obtained by performing target detection on the image to be detected; and the matching of each cluster with the initial matching point corresponding to each cluster to obtain the plurality of feature point matching pairs includes: If any cluster is not empty, matching is performed based on the distance data between the first feature point in any cluster and the initial matching point to obtain a feature point matching pair of at least part of the first feature points in any cluster.
5. The method according to claim 3, characterized in that: The first feature point set includes first feature points obtained by performing target detection on the image to be detected; and the matching of each cluster with the initial matching point corresponding to each cluster to obtain the plurality of feature point matching pairs includes at least one of the following situations: If any cluster is empty, no feature point matching pair is generated; If there is a first feature point in any cluster, a matching feature point matching the first feature point is determined from the initial matching points corresponding to the cluster, and a feature point matching pair is formed accordingly; If there are multiple first feature points in any cluster, and any cluster corresponds to an initial matching point, a first feature point matching the initial matching point is determined from the multiple first feature points, and a feature point matching pair is formed accordingly; If there are multiple first feature points in any cluster, and any cluster corresponds to multiple initial matching points, for any first feature point in any cluster, the matching feature points that match the any first feature point are determined from the initial matching points corresponding to the any cluster, and the corresponding feature point matching pairs constitute the any first feature point.
6. The method according to any one of claims 1 to 5, characterized in that: The correcting the image to be detected based on the matching of the multiple feature points to obtain a corrected detection image includes: Performing affine transformation based on the plurality of feature point matching pairs to obtain an affine transformation matrix; The image to be detected is corrected using the affine transformation matrix to obtain a corrected detection image.
7. The method according to any one of claims 1 to 5, characterized in that: The first feature point set is obtained by: Performing target detection on the image to be detected to obtain position data of multiple detection frames; Determine the center point of each detection frame based on the position data of each detection frame; The first feature point set is formed based on the center point of each detection frame.
8. The method according to any one of claims 1 to 5, characterized in that: The target feature points are determined by: Correcting the first feature point set according to a coordinate deviation between a first feature point in the first feature point set and a second feature point in the second feature point set to obtain a corrected feature point set; Determine, based on the distance data between the third feature point and the second feature point in the corrected feature point set, the feature points to be removed that meet the preset distance screening condition in the corrected feature point set; According to the revised feature point set and the second feature point set, it is determined whether to delete the feature points to be removed from the revised feature point set, so as to determine the target feature points that need to be retained in the first feature point set.
9. The method according to claim 8, characterized in that The determining whether to delete the feature points to be removed from the revised feature point set according to the revised feature point set and the second feature point set includes: Try to delete the feature points to be removed from the corrected feature point set to obtain a post-deletion feature point set; Determining, in the second feature point set, verification feature points corresponding to each feature point in the deleted feature point set to obtain a verification feature point set; If the sum of the distance deviations between the deleted feature point set and the verified feature point set is less than the deviation and threshold, determining to delete the feature point to be removed; If the sum of the distance deviations between the deleted feature point set and the verified feature point set is not less than the deviation sum threshold, it is determined that the feature point to be removed is not deleted.
10. An image correction device, characterized in that: The device comprises: A feature point matching module is used to obtain a first feature point set of the image to be detected and a second feature point set of the reference image to perform feature point matching to obtain target feature points in the first feature point set that match the second feature point set; a distribution determination module, configured to determine the distribution of the second feature points in the second feature point set on the reference image if the number of the target feature points is less than a preset number threshold; A feature point pairing module, configured to perform clustering and feature point pairing according to the distribution, the first feature point set, and the second feature point set to obtain a plurality of feature point matching pairs; wherein the number of the feature point matching pairs is greater than the preset number threshold; The image correction module to be detected is used to correct the image to be detected based on the matching of the multiple feature points to obtain a corrected detection image.
11. A computer device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to execute a computer program in the memory, and the computer program can execute the method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that: Used to store a computer program, which can execute the method according to any one of claims 1 to 9.
13. A chip comprising a storage unit and a processing unit, wherein the storage unit stores a computer program, characterized in that: When the processing unit executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.
Citation Information
Cited By
Data processing method and related device
CN121708098A
