Positioning model training method, system, device and readable storage medium
By generating and correcting pre-trained data sets, combining data augmentation and 360° coding technology, the ease of use and angle regression of existing template positioning algorithms in complex scenarios is solved, and efficient positioning is achieved in the case of few samples. Model training and accuracy.
Patent Information
- Application Number
- CN202411004986.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-25
AI Technical Summary
The existing template positioning algorithm needs to adjust a large number of hyperparameters in complex scenarios, which are poorly ease-of-use, and it is difficult to balance mismatch and mismatch, so it is impossible to construct a good positioning model, especially in terms of periodicity and discontinuity of the angle range. .
By generating pretrained data sets, filtering and correcting exception labeling data, training pretrained models using data augmentation technology, combining local data for positioning model training, and using 360° encoding and cosine phase decoding to improve the continuity and accuracy of angle regression .
Maintaining good prediction results with fewer samples improves the accuracy of complex data positioning, shortens positioning time, saves data labeling costs, and improves the accuracy of angle regression.
Smart Images

Figure CN118537684B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of machine vision image processing, and specifically relates to a positioning model training method, system, device and readable storage medium. Background Art
[0002] Template positioning is widely used in scenarios such as target recognition, component positioning, and region of interest extraction. It is particularly widely used in the field of industrial visual quality inspection. For example, when inspecting a large device, a positioning algorithm is often used to identify and locate the position of components, and whether the components are missing is determined based on the recognition results; the positioning results are often used to obtain the initial position of certain edges, and then the specific edges of the object are captured as a measurement reference; the positioning results and their relative position relationship are often used to intercept the region of interest to be identified, or a mask is constructed based on the positioning results to eliminate interference.
[0003] Although template matching based on existing algorithms has been widely used, when using traditional algorithms for positioning in complex scenes, a large number of hyperparameters need to be adjusted, and the algorithm is not easy to use. In addition, in some scenes, the existing algorithms cannot balance mismatches and missed matches by adjusting hyperparameters alone, and cannot construct a good positioning model.
[0004] Furthermore, existing algorithms require more training data, and most angle ranges are limited. The periodicity and discontinuity of angles cannot be solved.
[0005] Therefore, in view of the above technical problems, it is necessary to provide an improved positioning model training method.
[0006] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention
[0007] The object of the present invention is to provide a positioning model training method, system, device and readable storage medium, which can obtain better prediction results with fewer samples while avoiding a large number of hyper-parameter adjustments.
[0008] In order to achieve the above object, a technical solution provided by a specific embodiment of the present invention is as follows:
[0009] In a first aspect, the present invention provides a positioning model training method, which comprises:
[0010] Generate a pre-training data set, the pre-training data set comprising: a given image containing a target object and positioning annotation data corresponding to the given image, the positioning annotation data comprising vertex coordinates of a positioning box used to locate the target object;
[0011] Screening and correcting abnormal positioning and annotation data in the pre-training data set;
[0012] Performing data augmentation on the corrected pre-training data set, and training a pre-training model based on the augmented pre-training data set, and recording the pre-training model weight when the accuracy of the pre-training model reaches a preset accuracy threshold;
[0013] The pre-trained model weights are input into the local model, and the local model is trained based on a preset loss function to generate a positioning model, wherein the loss function includes a positioning box sine and cosine loss function.
[0014] In one or more embodiments of the present invention, a method for screening and correcting abnormal positioning and annotation data in the pre-training data set includes:
[0015] Obtaining the similarity between each positioning frame area and the target object, arranging the given image from small to large according to the similarity and dividing it into a first interval to an nth interval in order of arrangement, where n∈N+ and n>2;
[0016] Detect whether there is an abnormality in the positioning annotation data corresponding to the given image in the mth interval, m∈N+ and m≤n;
[0017] If so, check whether there is any abnormality in the positioning and annotation data in the m+1th interval;
[0018] If not, check whether the positioning and annotation data in the m-1th interval is abnormal;
[0019] When two adjacent intervals are detected, one of which has abnormal positioning and annotation data and the other has no abnormal positioning and annotation data, the detection is stopped, and the given image in the interval whose average similarity is less than or equal to the interval with abnormal positioning and annotation data in the two adjacent intervals is marked as the image to be processed.
[0020] In one or more embodiments of the present invention, the method further comprises:
[0021] Based on the preset sampling ratio, randomly select the given image and the corresponding positioning annotation data within the interval to be detected;
[0022] If the extracted positioning and annotation data does not contain any abnormality, it is deemed that the positioning and annotation data in the interval to be detected does not contain any abnormality.
[0023] In one or more embodiments of the present invention, a method for screening and correcting abnormal positioning and annotation data in the pre-training data set includes:
[0024] Obtain the parameter label of the image to be processed, the rotation overlap angle of the image to be processed and the vertex coordinates of the positioning frame corresponding to the image to be processed. If the image to be processed is a rotationally symmetric image, the rotation overlap angle is the rotation angle of the image to be processed. If the image to be processed is not a rotationally symmetric image, the rotation overlap angle is 360°.
[0025] Based on the parameter label of the given image, the rotation overlap angle of the given image and the vertex coordinates of the positioning frame, the corrected positioning annotation data of the image to be processed is generated.
[0026] In one or more embodiments of the present invention, the method further comprises:
[0027] If the given image in the pre-training data set has no corresponding positioning annotation data and is corrected, then the parameter label corresponding to the given image is set as the first parameter;
[0028] If the positioning and annotation data in the pre-training data set is wrong and has been corrected, setting the corresponding given image parameter label as the second parameter;
[0029] Set other given image parameter tags as the third parameter.
[0030] In one or more embodiments of the present invention, the method further comprises:
[0031] Train the test model based on the modified pre-training dataset;
[0032] Using a given image in the modified pre-training data set as an input of the verification model to generate verification annotation data;
[0033] Calculate the rotation intersection-over-combination ratio and the angle deviation of the positioning annotation data in the modified pre-training data set and the verification annotation data;
[0034] Based on the preset joint interval, it is determined whether the positioning and annotation data in the modified pre-training data set is abnormal, and the abnormal positioning and annotation data is corrected.
[0035] In one or more embodiments of the present invention, the method further comprises:
[0036] Based on the local image data, generate corresponding local positioning annotation data;
[0037] Performing data augmentation on the local image data and the corresponding local positioning annotation data to form local training data;
[0038] The local model is trained based on the local training data to generate a positioning model.
[0039] In one or more embodiments of the present invention, the method of inputting the pre-trained model weights into the local model and training the local model based on a preset loss function includes:
[0040] Solidifying the model feature extraction weights based on the pre-trained model weights;
[0041] The feature fusion and detection head of the local model are trained based on the local training data to generate a positioning model.
[0042] In one or more embodiments of the present invention, the loss function is:
[0043]
[0044] Among them, loss is the loss function, is the true value of the rotation phase of the positioning frame in N samples, is the predicted value of the rotation phase of the positioning frame in N samples, is the classification loss function, is the weight of the classification loss function, is the rotation box loss function, is the weight of the rotation box loss function, is the phase loss function, is the weight of the sinusoidal phase loss function, is the weight of the cosine loss function.
[0045] In one or more embodiments of the present invention, the method further comprises:
[0046] Define a vertex of the positioning frame as the starting vertex, and define the other vertices as the first vertex, the second vertex and the third vertex in a clockwise or counterclockwise direction;
[0047] Based on the coordinates of the starting vertex and the first vertex, obtaining a vector pointing from the starting vertex to the first vertex;
[0048] The angle between the vector from the starting vertex to the first vertex and the positive direction of the X-axis in the coordinate system is calculated, and the angle is the rotation angle of the positioning frame.
[0049] In a second aspect, the present invention provides a positioning method, which is applied to a positioning model obtained by the positioning model training method, and comprises:
[0050] Input the image to be located and the target object, perform feature extraction and feature fusion on the image to be located and the target object, and input the result into the detection head;
[0051] Generate the distance between the center point of the positioning frame and each side of the positioning frame, the position of the positioning frame, and the sine and cosine phases corresponding to the rotation angle of the positioning frame based on the detection head;
[0052] Based on a preset angle decoding formula, the sine and cosine phases are decoded into a positioning frame rotation angle value to complete the regression prediction of the positioning frame.
[0053] In one or more embodiments of the present invention, the angle decoding formula is:
[0054]
[0055] Among them, x is the cosine phase corresponding to the rotation angle of the positioning frame, and y is the sine phase corresponding to the rotation angle of the positioning frame. The decoded positioning frame rotation angle value.
[0056] In a third aspect, the present invention provides a positioning model training system, comprising:
[0057] A generation module, used to generate a pre-training data set, the pre-training data set comprising: a given image containing a target object and positioning annotation data corresponding to the given image, the positioning annotation data comprising vertex coordinates of a positioning box used to locate the target object;
[0058] A correction module, used to screen and correct abnormal positioning and annotation data in the pre-training data set;
[0059] A pre-training module, used to perform data augmentation on the modified pre-training data set, and train a pre-training model based on the augmented pre-training data set, and record the pre-training model weight when the accuracy of the pre-training model reaches a preset accuracy threshold;
[0060] A training module is used to input the pre-trained model weights into a local model, train the local model based on a preset loss function, and generate a positioning model. The loss function includes a positioning box sine and cosine loss function.
[0061] In a fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the positioning model training method by executing the computer instructions.
[0062] In a fifth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the positioning model training method.
[0063] Compared with the prior art, the positioning model training method provided by the present invention trains a pre-trained model with better feature extraction capabilities by positioning big data, and obtains the pre-trained model weights with a certain accuracy before local training, which can maintain good training effects even when there are fewer local samples.
[0064] The screening, correction and verification process of the training data ensures the accuracy of the data involved in the model training. Furthermore, the screening method provided by the present invention saves data annotation costs through the strategy of partition screening and spot check cleaning.
[0065] The present invention does not need to construct a complex search space on a complex data set. It only needs to add the complex data of scaling, occlusion, and blur to the training set, or generate approximate data for possible situations through data augmentation and add them to the training set. The feature extraction and recognition are completed end-to-end through the training model, which greatly improves the accuracy of complex data positioning and shortens the positioning time.
[0066] In the specific model training process, the present invention provides a 360° encoded rotating frame detection model, which can better express the positioning results. In addition, the improvement of the angle regression head avoids the discontinuity of the direct regression angle. The present invention uses the sine and cosine phase and atan2 decoding method to regress the rotating frame angle, taking into account its continuity and periodicity, and greatly improves the accuracy of angle regression at the ±π angle mutation position. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0068] Figure 1 is a schematic diagram of a positioning model training method in one embodiment of the present invention;
[0069] Figure 2 is a schematic diagram of a positioning model training system in one embodiment of the present invention;
[0070] Figure 3 is a structural block diagram of an electronic device in one embodiment of the present invention;
[0071] Figure 4 It is a data annotation schematic diagram of a specific embodiment of the present invention;
[0072] Figure 5 is a schematic diagram of sine and cosine phases and radian angles in one embodiment of the present invention;
[0073] Figure 6 is a schematic diagram of a positioning model in a specific embodiment of the present invention;
[0074] Figure 7 It is a schematic diagram of a positioning model detection head in a specific embodiment of the present invention. DETAILED DESCRIPTION
[0075] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.
[0076] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.
[0077] There are methods for template positioning in the existing technical solutions, and the existing algorithms mainly include grayscale-based algorithms and feature-based algorithms. Grayscale-based algorithms often use similarity evaluation indicators such as normalized cross-correlation, mutual information, and structural similarity (SSIM) to obtain the best matching position or the matching position with a similarity greater than a certain threshold based on the evaluation indicators. Feature-based algorithms often extract the edge features of the template, and then obtain their similarity through generalized Hough transform. Some algorithms also extract key point features and then calculate their descriptor similarity. Finally, based on the similarity evaluation indicators of the structural features, the template and the original image are obtained. The best matching position or the similarity is greater than a certain threshold. This type of algorithm often uses the pyramid method to increase the matching speed.
[0078] However, since a large number of hyperparameters need to be adjusted when using traditional algorithms for positioning in complex scenes, this technical solution will inevitably have the problem of poor algorithm usability. For example, if the scale of the template to be matched changes greatly or there is a perspective transformation, the search space will increase, the number of templates will increase, and the matching time will increase. In addition, there are some scenes where only adjusting the hyperparameters cannot balance the mismatch and missed match, and a good positioning model cannot be constructed. At the same time, the problems of angle periodicity and continuity are difficult to overcome.
[0079] On the other hand, most angle encoding methods in the prior art are long-side representation, that is, the angle between the long side of the positioning frame and the X-axis of the coordinate system is used as the rotation angle of the positioning frame. Since the positioning frame is rectangular, the angle range is only 0° to 180°, which also makes it impossible to distinguish the head and tail of the object.
[0080] The inventor of the present invention has found the main shortcomings of the prior art, and based on the shortcomings of the prior art, has proposed a new technical implementation idea: first, a given image in a large range is annotated, and the accuracy of the annotation is improved by gradually screening out abnormally annotated images, thereby forming a high-precision pre-trained model. The weights of the pre-trained model are input into the local model, and the corresponding sine and cosine loss functions are set, etc., and the local data is combined for training to generate a positioning model. Before local training, the weights of the pre-trained model with a certain accuracy are obtained, and a good training effect can be maintained even when there are fewer local samples. The accuracy of complex data positioning is greatly improved, and the positioning time is shortened. In the specific model training process, the present invention provides a 360° encoded rotating frame detection model, which can better express the positioning results. The improvement of the angle regression head avoids the discontinuity of the direct regression angle, takes into account its continuity and periodicity, and greatly improves the accuracy of angle regression at the ±π angle mutation position.
[0081] Please refer to Figure 1 FIG. 1 is a flow chart of a positioning model training method according to an embodiment of the present invention. The positioning model training method specifically comprises the following steps:
[0082] S101: Generate a pre-training data set, where the pre-training data set includes: a given image containing a target object and positioning annotation data corresponding to the given image.
[0083] It should be noted that the positioning annotation data is the annotation data of the target object on a given image. The annotation data may include but is not limited to: vertex coordinates of the corresponding positioning box, given image rotation overlap angle, parameter labels, etc., or it may be only one or more of them.
[0084] In an embodiment of the present invention, the method for generating the pre-training data set of the present invention may include but is not limited to: generating annotations for a given image based on region matching or generating annotations for a given image based on feature matching, etc., and the embodiment of the present invention does not limit this.
[0085] For example, based on region matching, a specific object in a given image can be relatively accurately identified and located, and the specific object can be labeled, which is suitable for situations where the target object lacks significant features in the image. By sliding the target image to compare with the given image, the target image moves on the given image pixel by pixel, and a match is found by calculating the similarity value of the overlapping area.
[0086] Feature matching is applicable to the case where the target image has a high consistency with the given image in terms of features and control points. In this case, the features can be points, curves, or surface models, which are used to perform template matching. The corresponding steps are: find a set of unique key points; define a region for each key point; extract and standardize the region content; calculate local descriptors from the standardized region; match local descriptors.
[0087] It should also be noted that after the corresponding given image is marked by any of the above matching methods, the similarity between the corresponding target object and the given graphic positioning area can be obtained based on the matching result.
[0088] S102: Screening and correcting abnormal positioning and annotation data in the pre-training data set;
[0089] It should be noted that although the pre-training data set generated based on target image matching has the advantages of good matching consistency and high positioning accuracy, it cannot balance the situations of mismatching and missed matching in complex scenarios. For example, to ensure the accuracy of the positioning model, the pre-training data generated by the template matching algorithm should not be used directly for model training. It is necessary to screen and correct the abnormal positioning annotation data in the pre-training data set.
[0090] In an exemplary embodiment, screening abnormal positioning and annotation data in a pre-training data set includes: obtaining the similarity between each positioning frame area and the target object, arranging the given image in ascending order according to the similarity and dividing it into the first interval to the nth interval in order of arrangement, where n∈N+ and n>2; detecting whether the positioning and annotation data corresponding to the given image in the mth interval has an abnormality, m∈N+ and m≤n; if so, detecting whether the positioning and annotation data in the m+1th interval has an abnormality; if not, detecting whether the positioning and annotation data in the m-1th interval has an abnormality; when it is detected that two adjacent intervals have abnormal positioning and annotation data in one and no abnormal positioning and annotation data in the other, stopping the detection, and marking the given image in the interval whose average similarity is less than or equal to the interval with abnormal positioning and annotation data in the two adjacent intervals as the image to be processed.
[0091] In an embodiment of the present invention, it is believed that the similarity is positively correlated with the probability of abnormal positioning annotation. Therefore, when an annotation abnormality occurs in an interval, it can be approximately determined that the positioning annotation data in all intervals whose average similarity is less than or equal to the interval are unreliable, and the positioning annotation needs to be cleaned and re-annotated. On the other hand, when no annotation abnormality occurs in an interval, that is, the data in the interval is credible, it can be approximately determined that the positioning annotation data in all intervals whose average similarity is greater than or equal to the interval are credible.
[0092] It should be noted that the abnormal positioning annotation data may include positioning annotation data errors or a given image without corresponding positioning annotation data. For missing positioning data, that is, a given image without corresponding positioning annotation data, since there is no corresponding record of similarity, its similarity is defaulted to the lowest to ensure effective screening of missing positioning data. On the other hand, the embodiment of the present invention does not limit the interval division method based on similarity and the selection of the starting inspection interval.
[0093] For the selection of the starting test interval, it is preferred to use the interval with the average similarity in the middle as the starting test interval. Since the complexity of the usage scenario will affect the credibility of the generated pre-training data set, the selection of the starting test interval can also be adaptively adjusted based on different actual usage scenarios.
[0094] Regarding the way of dividing the intervals, in an exemplary embodiment, in order to facilitate the interval setting and subsequent screening operations, the above n intervals can be set equidistantly; in another exemplary embodiment, the positioning data with low similarity to the preset template has a greater probability of labeling anomalies, and vice versa, the positioning data with high similarity to the preset template has a lower probability of labeling anomalies. In order to make the screening more accurate, the width of the corresponding interval can be gradually reduced from small to large similarity. It can be understood that the interval division method and the selection of the starting inspection interval are not limited to the above types. Those skilled in the art should be able to infer other embodiments based on the description in the specification and in combination with specific usage scenarios.
[0095] It should also be noted that in the above-mentioned process of screening the pre-training data set, if there are multiple categories of given images or target images, the given images can also be grouped before sorting them based on similarity. The given images in each group are sorted and screened separately. Or based on a preset sampling ratio, randomly extract the given images and the corresponding positioning annotation data in the interval to be detected; if there is no abnormality in the extracted positioning annotation data, it is deemed that there is no abnormality in the positioning annotation data in the interval to be detected.
[0096] The embodiment of the present invention does not limit whether to perform additional means such as grouping or sampling for auxiliary judgment, nor does it limit the grouping standard and sampling logic, etc. The direct or indirect application of the above screening method should be included in the protection scope of the present invention.
[0097] Furthermore, the abnormal data is relabeled, that is, the abnormal positioning and labeling data in the pre-training data set is corrected, including but not limited to: obtaining the parameter label of the image to be processed, the rotation overlap angle of the image to be processed and the vertex coordinates of the positioning box corresponding to the image to be processed. If the image to be processed is a rotationally symmetric image, the rotation overlap angle is the rotation angle of the image to be processed. If the image to be processed is not a rotationally symmetric image, the rotation overlap angle is 360°; based on the parameter label of a given image, the rotation overlap angle of a given image and the vertex coordinates of the positioning box, etc., the corrected positioning and labeling data of the image to be processed is generated.
[0098] The parameter label of a given image is used to characterize the state of the corresponding positioning annotation information, specifically including: the given image has no corresponding positioning annotation data and is corrected, the positioning annotation data is wrong and is corrected, and other data. Data of different states correspond to different parameter labels, and data of the same state correspond to the same parameter label. The positioning box is a rectangular box used to describe the position and range of the target in the image. The sides of the positioning box can form a certain angle with the coordinate axis of the plane coordinate system where the graphic is located, that is, the sides of the positioning box are not required to be perpendicular to the coordinate axis.
[0099] For example, Figure 4 As shown, it is a schematic diagram of a specific embodiment of the present invention. In this embodiment, it includes a given image A, a given image B, and a given image C. The given image A is a rectangle with an arrow pattern, the given image B is a rectangle without texture, and the given image C is a square without texture. In this embodiment, the given image A is a non-rotationally symmetrical figure, so its rotation overlap angle is 360°; the given image B and the given image C are rotationally symmetrical figures, so their rotation angles are their rotation overlap angles, corresponding to 180° for the given image B and 90° for the given image C.
[0100] Taking the given image A whose positioning annotation data is wrong and corrected as an example, if the present embodiment sets the given image to have no corresponding positioning annotation data and corrected, the corresponding given image parameter label is set to 1; if the positioning annotation data is wrong and corrected, the corresponding given image parameter label is set to -1; and the other given image parameter labels are set to 0. The parameter label of the given image A should be set to -1 accordingly. Combined with other information of the given image A, the corresponding corrected label can be obtained as "x1, y1, x2, y2, x3, y3, x4, y4, 360°, -1....".
[0101] Since the present invention is different from the traditional method and adopts a 360° encoding method, the angle range of the rotationally symmetrical figure can be determined based on the setting of the rotation coincidence angle, so that it also uses the above-mentioned 360° encoding method. In one embodiment of the present invention, the center of the image in the positioning area points to a vertex of the image to form a vector. The angle of the positioning frame is the angle between the positive direction of the X-axis in the preset coordinate system and the vector, and is sorted by priority, preferably located in the first quadrant, the second quadrant, the third quadrant, and the fourth quadrant.
[0102] Using the above embodiment, given that the rotation overlap angle of image A is 360 degrees, O1 is located at the center of the object, O1X1 points to the vertex of the object, X1O1X is the rotation angle of the labeled object, and the range of the rotation angle is For a given image B, when annotating, O1X1 points to the vertex of the short side of the object, preferably in the first quadrant, and secondly in the second quadrant. This type of data is similar to the long side representation, with a rotation angle of X2O2X and an angle range of [0°, 180°). For a given image C, the rotation angle is O3X3 pointing to the short side, and preferably in the first quadrant, with a rotation angle of X3O3X and an angle range of [0°, 90°).
[0103] It should be noted that, in order to facilitate the generation of labels, in one embodiment of the present invention, the rotation overlap angle can be converted into a label value, and the label value can be included in the positioning label data and the corrected label instead of the angle value.
[0104] The conversion formula can be:
[0105]
[0106] in, is the rotation overlap angle value, and ns is the annotation value included in the positioning label data and the corrected label after conversion.
[0107] In order to further improve the accuracy of the pre-training data set, the positioning model training method provided by the present invention also includes: training a verification model based on a modified pre-training data set; using a given image in the modified pre-training data set as an input of the verification model to generate verification annotation data; calculating the rotational intersection ratio and angle deviation of the positioning annotation data in the modified pre-training data set and the verification annotation data; based on a preset joint interval, determining whether the positioning annotation data in the modified pre-training data set is abnormal, and correcting the abnormal positioning annotation data.
[0108] S103: performing data augmentation on the corrected pre-training data set, and training a pre-training model based on the augmented pre-training data set, and when the accuracy of the pre-training model reaches a preset accuracy threshold, recording the weight of the pre-training model;
[0109] It should be noted that data augmentation, also known as data enhancement, is a regularization method that can perform a series of random changes on the data to generate similar but different samples, thereby expanding the size of the data set. This method aims to improve the generalization ability of the model by increasing the amount and diversity of the training data set in scenarios where the amount of data is insufficient or the model parameters are large.
[0110] Model training based on the augmented pre-training data set has better results and higher accuracy. When the accuracy of the pre-training model reaches the preset accuracy threshold, the weight information of the model at this time can be recorded, which facilitates the training of the final positioning model.
[0111] It should be noted that the embodiments of the present invention do not limit the specific data augmentation methods used. Since the pre-training data set has more data samples than the local training data set, more information needs to be processed for model training. Therefore, in order to reduce the time to the preset accuracy and reduce the burden on the system, under the premise of ensuring the training accuracy of the pre-training model, it is preferred that only data augmentation methods related to spatial position and size are used in the pre-training stage. In contrast, in the local training process, since the amount of data is lower than the pre-training data set, and in order to pursue higher accuracy, it is preferred to add offline data augmentation while augmenting the data related to spatial position and size.
[0112] It should also be noted that in order to avoid a large amount of identical redundancy in the augmentation of rotationally symmetric figures, when augmenting a given image, it is necessary to read the positioning mark data corresponding to the given image to obtain its rotation overlap angle β° or ns value. The augmentation angle range is determined to be (0°, β°).
[0113] In an exemplary embodiment, the above-mentioned offline data augmentation method can use the getRotationMatrix2D function in OpenCV, with the center of the marked rotation box as the rotation center, [-180°, 180°) as the angle range, and a step size of 5° to generate a rotation transformation matrix, transform the corresponding image and label, and convert the label angle to an angle range matching ns. This augmentation method obtains augmented data of each angle of the template, a priori increases the diversity of training data angles, and avoids the problem that the rotation center of imgaug rotation augmentation is random, and it is difficult to augment templates of various angles.
[0114] It can be understood that the data augmentation method used in the embodiments of the present invention is not limited.
[0115] S104: Input the pre-trained model weights into the local model, train the local model based on a preset loss function, and generate a positioning model, wherein the loss function includes a positioning box sine and cosine loss function.
[0116] In an exemplary embodiment of the present invention, the loss function is:
[0117]
[0118] Among them, loss is the loss function, is the true value of the rotation phase of the positioning frame in N samples, is the predicted value of the rotation phase of the positioning frame in N samples. is the classification loss function, is the weight of the classification loss function, is the rotation box loss function, is the weight of the rotation box loss function, is the phase loss function, is the weight of the sinusoidal phase loss function, is the weight of the cosine loss function.
[0119] It should be noted that the classification loss function preferably adopts the Focal Loss loss function, which has a better effect on the case of category imbalance. The rotation box loss function preferably adopts the RotatedIoULoss loss function. The embodiment of the present invention does not limit the selection of a specific loss function.
[0120] Specifically, the Focal Loss loss function is:
[0121]
[0122] is the balance factor, To calculate the modulation factor, is the probability value associated with the predicted probability.
[0123] The RotatedIoULoss loss function is:
[0124]
[0125] It is assumed that the area of the predicted positioning box is PBox, the area of the annotated positioning box is GBox, and the intersection area of the predicted positioning box and the annotated positioning box is IBox.
[0126] Similar to the aforementioned pre-trained model, training the local model to generate a positioning model also includes: generating corresponding local positioning annotation data based on local image data; augmenting the local image data and the corresponding local positioning annotation data to form local training data; and training the local model based on the local training data to generate a positioning model.
[0127] Please refer to Figure 6, which is a structural diagram of the positioning model in a specific embodiment of the present invention. The positioning model proposed in the present invention mainly includes three modules: BackBone (feature extraction module), Neck (feature fusion module), and Head (detection head module). The model weights with better retention results are tested on a large number of pre-training data sets as pre-training weights, so that the feature extraction module of the model has good feature extraction capabilities. Therefore, in an exemplary embodiment, training the local model based on a preset loss function includes: solidifying the model feature extraction weights based on the pre-training model weights; training the feature fusion and detection head of the local model based on the local training data to generate a positioning model.
[0128] In addition, in order to achieve 360° encoding, that is, the obtained rotation frame angle is (-180°, 180°], the positioning model training method provided by the present invention also includes: defining a vertex of the positioning frame as the starting vertex, and along the clockwise or counterclockwise direction, the other vertices are the first vertex, the second vertex and the third vertex respectively; based on the coordinates of the starting vertex and the first vertex, obtaining the vector from the starting vertex to the first vertex; calculating the angle between the vector from the starting vertex to the first vertex and the positive direction of the X-axis in the coordinate system, and the angle is the rotation angle of the positioning frame.
[0129] It is understandable that based on the vertex coordinates, the rotation angle of the positioning frame, the center coordinates of the positioning frame, the length and width of the positioning frame and other data can be determined within a range of 360°. Therefore, recording the vertex coordinates in the positioning tag data can greatly simplify the data type of the tag, making it easier to learn and set.
[0130] The present invention also provides a positioning method, which is applied to the positioning model obtained by the above-mentioned positioning model training method, and specifically includes: inputting the image to be positioned and the target object, performing feature extraction and feature fusion on the image to be positioned and the target object, and inputting the result into the detection head; based on the detection head, generating the distance between the center point of the positioning frame and each side of the positioning frame, the positioning frame position and the sine and cosine phases corresponding to the rotation angle of the positioning frame; based on a preset angle decoding formula, decoding the sine and cosine phases into the rotation angle value of the positioning frame, and completing the regression prediction of the positioning frame.
[0131] Please refer to Figure 5As shown in the figure, it is a schematic diagram of sine and cosine phase and radian angle. It can be seen from the figure that in the existing algorithm, the angle prediction value of the rotating frame is discontinuous. The traditional algorithm detection head directly regresses the angle, where -π and π angles are consistent, but the regression value distance differs by 2π. This method leads to poor angle regression effect near ±π. The present invention improves it to sine and cosine phase representation sin(x), cos(x), which are both continuous periodic functions with a period of 2π. Near ±π, sin(x) is close to 0 and cos(x) is close to -1. It has a continuous representation, which can take into account continuity and periodicity, and greatly improves the accuracy of angle regression at the ±π angle mutation position.
[0132] For example, see Figure 6 , which is a structural diagram of a positioning model in a specific embodiment of the present invention.
[0133] The positioning model proposed in the present invention mainly includes three modules: BackBone (feature extraction module), Neck (feature fusion module), and Head (detection head module).
[0134] BackBone and Neck are similar to traditional single-stage rotation box detection. For example, after the input BatchSize, Channel, Height, and Width are 1, 3, 640, and 640 respectively, the feature tensors obtained by the convolutional neural network BackBone are C3, C4, and C5 respectively. Their sizes are 1×256×80×80, 1×512×40×40, and 1×1024×20×20 respectively as shown in the figure. Then, after the Neck (PAFPN) fusion features, the fusion feature outputs O3, O4, and O5 at different scales can be obtained. The tensor sizes are 1×256×80×80, 1×256×40×40, and 1×256×20×20 respectively.
[0135] Among them, R stands for ReduceLayer, U stands for UpSampleLayer, C stands for ConcatLayer, T stands for TopDownLayer, D stands for DownSampleLayer, B stands for BottomUpLayer, and O stands for OutLayer.
[0136] Specifically, BatchSize is the total number of training samples input in a single batch, and Channel is the number of color channels in a given image. For color images, the number of channels is 3 (red, green, and blue). For monochrome images, there is only one channel (black). Height and Width are the length and width of the corresponding given image.
[0137] Please refer to Figure 7, which is a schematic diagram of the detection head in a specific embodiment of the present invention. The ConvModule (convolution block) of the same Head shares convolution, but uses independent batch norm, and finally obtains the corresponding output result through the corresponding conv2d convolution. For example, after O3 inputs the Head, the Cls Head tensor size is 1×n×80×80, where n represents the number of categories, and the corresponding result is the confidence of whether a certain category exists at this position; the RotateBox Head tensor size is 1×4×80×80, and the corresponding result is expressed as the distance from the center point of the positioning box to the four sides, which is decoded into a rotated rectangular box in combination with the positioning box position and the positioning box rotation angle; Sin Phase and Cos Phase are the corresponding angle detection heads, which share the ConvModule module, use different Conv2d to obtain the sine and cosine phases, and finally obtain the corresponding angles through atan2 decoding, and their output tensors are all 1×1×80×80.
[0138] Specifically, the angle decoding formula is:
[0139]
[0140] Among them, x is the cosine phase corresponding to the rotation angle of the positioning frame, and y is the sine phase corresponding to the rotation angle of the positioning frame. The decoded positioning frame rotation angle value.
[0141] Please refer to Figure 2 As shown, based on the same inventive concept as the aforementioned positioning model training method, an embodiment of the present invention provides a positioning model training system 200, which includes: a generation module 201, a correction module 202, a pre-training module 203 and a training module 204.
[0142] Specifically, the generation module 201 is used to generate a pre-training data set, and the pre-training data set includes: a given image containing a target object and positioning annotation data corresponding to the given image, and the positioning annotation data includes the rotation angle and vertex coordinates of the positioning box used to locate the target object; the correction module 202 is used to screen and correct abnormal positioning annotation data in the pre-training data set; the pre-training module 203 is used to perform data augmentation on the corrected pre-training data set, and train a pre-training model based on the augmented pre-training data set, and when the accuracy of the pre-training model reaches a preset accuracy threshold, the pre-training model weight is recorded; the training module 204 is used to input the pre-training model weight into a local model, train the local model based on a preset loss function, and generate a positioning model, and the loss function includes a positioning box sine and cosine loss function.
[0143] It should be noted that the correction module 202 is also used to obtain the similarity between each positioning frame area and the target object, arrange the given image from small to large according to the similarity and divide it into the first interval to the nth interval in the order of arrangement, where n∈N+ and n>2; detect whether there is an abnormality in the positioning annotation data corresponding to the given image in the mth interval, m∈N+ and m≤n; if so, detect whether there is an abnormality in the positioning annotation data in the m+1th interval; if not, detect whether there is an abnormality in the positioning annotation data in the m-1th interval; when two adjacent intervals are detected, one of which has abnormal positioning annotation data and the other has no abnormal positioning annotation data, stop the detection, and mark the given image in the interval with an average similarity less than or equal to the interval with abnormal positioning annotation data in the two adjacent intervals as the image to be processed.
[0144] The correction module 202 is also used to obtain the parameter label of the image to be processed, the rotation overlap angle of the image to be processed, and the vertex coordinates and rotation angle of the corresponding positioning frame of the image to be processed. If the image to be processed is a rotationally symmetric image, the rotation overlap angle is the rotation angle of the image to be processed. If the image to be processed is not a rotationally symmetric image, the rotation overlap angle is 360°; based on the parameter label of the given image, the rotation overlap angle of the given image and the vertex coordinates of the positioning frame, the corrected positioning annotation data of the image to be processed is generated.
[0145] It should also be noted that the above-mentioned positioning model training system 200 also includes a sampling module, which is used to randomly extract a given image and corresponding positioning annotation data within the interval to be detected based on a preset sampling ratio; if the extracted positioning annotation data does not contain any abnormality, it is deemed that the positioning annotation data within the interval to be detected does not contain any abnormality.
[0146] The positioning model training system 200 also includes a setting module, which is used to set the corresponding given image parameter label as the first parameter when the given image in the pre-training data set has no corresponding positioning annotation data and is corrected; when the positioning annotation data in the pre-training data set is wrong and is corrected, set the corresponding given image parameter label as the second parameter; and set other given image parameter labels as the third parameter.
[0147] The positioning model training system 200 also includes a review module, which is used to train a verification model based on a modified pre-training data set; use a given image in the modified pre-training data set as input to the verification model to generate verification annotation data; calculate the rotational intersection ratio and angle deviation of the positioning annotation data in the modified pre-training data set and the verification annotation data; based on a preset joint interval, determine whether the positioning annotation data in the modified pre-training data set is abnormal, and correct the abnormal positioning annotation data.
[0148] The positioning model training system 200 also includes a processing module for generating corresponding local positioning annotation data based on local image data; performing data augmentation on the local image data and the corresponding local positioning annotation data to form local training data; and training the local model based on the local training data to generate a positioning model.
[0149] Please refer to Figure 3 As shown, an embodiment of the present invention further provides an electronic device 300, which includes at least one processor 301, a memory 302 (such as a non-volatile memory), a memory 303, and a communication interface 304, and the at least one processor 301, the memory 302, the memory 303, and the communication interface 304 are connected together via a bus 305. At least one processor 301 is used to call at least one program instruction stored or encoded in the memory 302, so that the at least one processor 301 performs various operations and functions of the positioning model training method described in various embodiments of this specification.
[0150] In the embodiments of the present specification, the electronic device 300 may include, but is not limited to, personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile electronic devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable electronic devices, consumer electronic devices, and the like.
[0151] An embodiment of the present invention also provides a computer-readable medium, which carries computer execution instructions. When the computer execution instructions are executed by a processor, they can be used to implement various operations and functions of the positioning model training method described in various embodiments of this specification.
[0152] The computer-readable medium in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device.
[0153] In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0154] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0155] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, systems, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0156] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.
Claims
1. A positioning model training method, characterized in that: include: Generate a pre-training data set, the pre-training data set comprising: a given image containing a target object and positioning annotation data corresponding to the given image, the positioning annotation data comprising vertex coordinates of a positioning box used to locate the target object; Screening and correcting abnormal positioning and annotation data in the pre-training data set; Performing data augmentation on the corrected pre-training data set, and training a pre-training model based on the augmented pre-training data set, and recording the pre-training model weight when the accuracy of the pre-training model reaches a preset accuracy threshold; Inputting the pre-trained model weights into the local model, training the local model based on a preset loss function to generate a positioning model, wherein the loss function includes a positioning box sine and cosine loss function; The positioning model training method also includes: training a verification model based on a modified pre-training data set; using a given image in the modified pre-training data set as input to the verification model to generate verification annotation data; calculating a rotational intersection-and-union ratio and an angle deviation between the positioning annotation data in the modified pre-training data set and the verification annotation data; and judging whether the positioning annotation data in the modified pre-training data set is abnormal based on a preset joint interval, and correcting the abnormal positioning annotation data.
2. The positioning model training method according to claim 1, characterized in that: The method for screening and correcting abnormal positioning and annotation data in the pre-training data set includes: Obtaining the similarity between each positioning frame area and the target object, arranging the given image from small to large according to the similarity and dividing it into a first interval to an nth interval in order of arrangement, where n∈N+ and n>2; Detect whether there is an abnormality in the positioning annotation data corresponding to the given image in the mth interval, m∈N+ and m≤n; If so, check whether there is any abnormality in the positioning and annotation data in the m+1th interval; If not, check whether the positioning and annotation data in the m-1th interval is abnormal; When two adjacent intervals are detected, one of which has abnormal positioning and annotation data and the other has no abnormal positioning and annotation data, the detection is stopped, and the given image in the interval whose average similarity is less than or equal to the interval with abnormal positioning and annotation data in the two adjacent intervals is marked as the image to be processed.
3. The positioning model training method according to claim 2, characterized in that: The method further comprises: Based on the preset sampling ratio, randomly select the given image and the corresponding positioning annotation data within the interval to be detected; If the extracted positioning and annotation data does not contain any abnormality, it is deemed that the positioning and annotation data in the interval to be detected does not contain any abnormality.
4. The positioning model training method according to claim 3, characterized in that: The method for screening and correcting abnormal positioning and annotation data in the pre-training data set includes: Obtain the parameter label of the image to be processed, the rotation overlap angle of the image to be processed and the vertex coordinates of the positioning frame corresponding to the image to be processed. If the image to be processed is a rotationally symmetric image, the rotation overlap angle is the rotation angle of the image to be processed. If the image to be processed is not a rotationally symmetric image, the rotation overlap angle is 360°. Based on the parameter label of the given image, the rotation overlap angle of the given image and the vertex coordinates of the positioning frame, the corrected positioning annotation data of the image to be processed is generated.
5. The positioning model training method according to claim 4, characterized in that: The method further comprises: If the given image in the pre-training data set has no corresponding positioning annotation data and is corrected, then the parameter label corresponding to the given image is set as the first parameter; If the positioning and annotation data in the pre-training data set is wrong and has been corrected, setting the corresponding given image parameter label as the second parameter; Set other given image parameter tags as the third parameter.
6. The positioning model training method according to claim 1, characterized in that: The method further comprises: Based on the local image data, generate corresponding local positioning annotation data; Performing data augmentation on the local image data and the corresponding local positioning annotation data to form local training data; The local model is trained based on the local training data to generate a positioning model.
7. The positioning model training method according to claim 6, characterized in that: The method of inputting the pre-trained model weights into a local model and training the local model based on a preset loss function includes: Solidifying the model feature extraction weights based on the pre-trained model weights; The feature fusion and detection head of the local model are trained based on the local training data to generate a positioning model.
8. The positioning model training method according to claim 1, characterized in that: The loss function is: Among them, loss is the loss function, is the true value of the rotation phase of the positioning frame in N samples, is the predicted value of the rotation phase of the positioning frame in N samples, is the classification loss function, is the weight of the classification loss function, is the rotation box loss function, is the weight of the rotation box loss function, is the phase loss function, is the weight of the sinusoidal phase loss function, is the weight of the cosine loss function.
9. The positioning model training method according to claim 1, characterized in that: The method further comprises: Define a vertex of the positioning frame as the starting vertex, and define the other vertices as the first vertex, the second vertex and the third vertex in a clockwise or counterclockwise direction; Based on the coordinates of the starting vertex and the first vertex, obtaining a vector pointing from the starting vertex to the first vertex; The angle between the vector from the starting vertex to the first vertex and the positive direction of the X-axis in the coordinate system is calculated, and the angle is the rotation angle of the positioning frame.
10. A positioning method, applied to a positioning model obtained by the positioning model training method according to any one of claims 1 to 9, characterized in that: include: Input the image to be located and the target object, perform feature extraction and feature fusion on the image to be located and the target object, and input the result into the detection head; Generate the distance between the center point of the positioning frame and each side of the positioning frame, the position of the positioning frame, and the sine and cosine phases corresponding to the rotation angle of the positioning frame based on the detection head; Based on a preset angle decoding formula, the sine and cosine phases are decoded into a positioning frame rotation angle value to complete the regression prediction of the positioning frame.
11. The positioning method according to claim 10, characterized in that: The angle decoding formula is: Among them, x is the cosine phase corresponding to the rotation angle of the positioning frame, and y is the sine phase corresponding to the rotation angle of the positioning frame. The decoded positioning frame rotation angle value.
12. A positioning model training system, characterized in that: include: A generation module, used to generate a pre-training data set, the pre-training data set comprising: a given image containing a target object and positioning annotation data corresponding to the given image, the positioning annotation data comprising vertex coordinates of a positioning box used to locate the target object; A correction module, used to screen and correct abnormal positioning and annotation data in the pre-training data set; A pre-training module, used to perform data augmentation on the modified pre-training data set, and train a pre-training model based on the augmented pre-training data set, and record the pre-training model weight when the accuracy of the pre-training model reaches a preset accuracy threshold; A training module, used for inputting the pre-trained model weights into a local model, training the local model based on a preset loss function, and generating a positioning model, wherein the loss function includes a positioning box sine and cosine loss function; The positioning model training system also includes: training a verification model based on a modified pre-training data set; using a given image in the modified pre-training data set as input to the verification model to generate verification annotation data; calculating a rotational intersection-and-union ratio and an angle deviation between the positioning annotation data in the modified pre-training data set and the verification annotation data; and judging whether the positioning annotation data in the modified pre-training data set is abnormal based on a preset joint interval, and correcting the abnormal positioning annotation data.
13. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the positioning model training method according to any one of claims 1 to 9 by executing the computer instructions.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the positioning model training method described in any one of claims 1-9.
Citation Information
Patent Citations
Method for segmenting to-be-detected region of pole piece based on Yolov5 algorithm
CN115908447A
Vehicle boundary positioning method for night scene
CN117152513A
