Quadrilateral object detection, model training method, device, equipment and storage medium

The neural network model outputs the target detection box and its vertex offset variable values ​​of the quadrilateral object, which solves the problem of poor tilt detection effect of the quadrilateral object and achieves accurate and stable detection effects at any tilt angle.

CN112183529BActive Publication Date: 2024-11-08CHUANGXIN QIZHI (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011012425.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-23
Publication Date
2024-11-08
Estimated Expiration
2040-09-23

AI Technical Summary

Technical Problem

The existing quadrilateral object detection method is poor when the object is inclined more than 45 degrees, and the detection effect is unstable when the inclination angle is particularly small, so accurate detection cannot be achieved.

Method used

By using a pre-trained neural network model, the object detection box containing the quadrilateral object and its four vertices are directly output, and the vertices of the object detection box are offset according to the offset variable value to obtain the quadrilateral object's Predicting vertices avoids the limitation that vertices must be on the edge of the detection box, so that the edges of the detection box no longer limit the vertex offset of the quadrilateral object.

Benefits of technology

Even if the quadrilateral object is inclined more than 45 degrees or the inclination angle is particularly small, the detection effect remains accurate and stable, and the accurate positioning of the quadrilateral object can be effectively achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112183529B_ABST
    Figure CN112183529B_ABST
Patent Text Reader

Abstract

The present application provides a quadrilateral object detection, model training method, device, equipment and storage medium. The detection method includes: obtaining an image to be detected containing a quadrilateral object to be detected; using a pre-trained neural network model to process the image to be detected, obtaining a target detection frame containing the quadrilateral object, and the offset variable values ​​corresponding to the four vertices of the target detection frame; offsetting each vertex of the target detection frame according to the offset variable value corresponding to each vertex, to obtain the predicted vertex of the quadrilateral object. Compared with the related art, the predicted vertex of the quadrilateral object obtained in this way is not restricted by the edge of the target detection frame, but is based on the four vertices of the target detection frame to predict the offset of the vertex of the quadrilateral object. Even if the quadrilateral object is tilted more than 45 degrees or the tilt angle of the quadrilateral object is particularly small, the detection effect can remain accurate and stable.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a quadrilateral object detection, model training method, device, equipment and storage medium. Background Art

[0002] In the field of image detection, it is often necessary to accurately locate quadrilateral objects in images (such as license plates, parking spaces, houses, Chinese sentences, etc.).

[0003] At present, the conventional detection method outputs a horizontal rectangular frame (i.e., detection frame) containing the quadrilateral object. In actual applications, the quadrilateral object is indeed horizontal in the picture, but there are also cases where the quadrilateral object is tilted in the picture. In this case, the output rectangular frame cannot coincide with the outline of the quadrilateral object, that is, it is impossible to accurately detect the quadrilateral material.

[0004] In this regard, the paper "Gliding Vertex on the Horizontal Bounding Box for Multi-Oriented Object Detection" proposes a solution that can achieve accurate positioning of quadrilateral objects. It first detects a large rectangular box (i.e., detection box) containing the quadrilateral object, and then re-determines the positions of the four vertices of the quadrilateral object by sliding clockwise on the edges formed by the four points of the rectangular box, thereby obtaining the quadrilateral box of the object.

[0005] However, this method does not work well when the quadrilateral object is tilted more than 45 degrees, and the detection effect is unstable when the tilt angle of the quadrilateral object is particularly small. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a quadrilateral object detection, model training method, device, equipment and storage medium to achieve position detection of quadrilateral objects.

[0007] An embodiment of the present application provides a method for detecting a quadrilateral object, comprising: acquiring an image to be detected; the image to be detected includes a quadrilateral object to be detected; using a pre-trained neural network model to process the image to be detected to obtain a target detection frame including the quadrilateral object, and offset variable values ​​corresponding to four vertices of the target detection frame; offsetting each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain a predicted vertex of the quadrilateral object.

[0008] In an embodiment of the present application, a target detection frame including a quadrilateral object and the offset variable values ​​corresponding to the four vertices of the target detection frame are directly outputted through a neural network model, and then each vertex of the target detection frame is offset according to the offset variable value corresponding to each vertex to obtain the predicted vertices of the quadrilateral object. Compared with the related art, the predicted vertices of the quadrilateral object obtained in this way are not restricted by the edges of the target detection frame, and the vertices of the quadrilateral object are not limited to the edges of the target detection frame. That is, the offset of the vertices of the quadrilateral object is not predicted based on the edges of the target detection frame, but based on the four vertices of the target detection frame, so that even when the quadrilateral object is tilted more than 45 degrees or the tilt angle of the quadrilateral object is particularly small, the detection effect can remain accurate and stable.

[0009] Furthermore, the vertex has an x-coordinate value and a y-coordinate value; the offset variable value corresponding to the vertex includes: an x-axis offset variable value corresponding to the x-coordinate value of the vertex, and a y-axis offset variable value corresponding to the y-coordinate value of the vertex; offsetting each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertices of the quadrilateral object, including: determining the width and height of the target detection frame; determining the x-coordinate predicted value of each predicted vertex of the quadrilateral object according to the width, the x-coordinate value of each vertex of the target detection frame and the corresponding x-axis offset variable value; determining the y-coordinate predicted value of each predicted vertex of the quadrilateral object according to the height, the y-coordinate value of each vertex of the target detection frame and the corresponding y-axis offset variable value.

[0010] Further, according to the width, the x-coordinate value of each vertex of the target detection frame and the corresponding x-axis offset variable value, the x-coordinate predicted value of each predicted vertex of the quadrilateral object is determined; and according to the height, the y-coordinate value of each vertex of the target detection frame and the corresponding y-axis offset variable value, the y-coordinate predicted value of each predicted vertex of the quadrilateral object is determined, including:

[0011] The predicted x-coordinate value and the predicted y-coordinate value of each predicted vertex of the quadrilateral object are determined according to the following formula:

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020] in:

[0021] x0 and y0 constitute the coordinates of a predicted vertex, x1 and y1 constitute the coordinate value of a predicted vertex, x2 and y2 constitute the coordinate value of a predicted vertex, and x3 and y3 constitute the coordinate value of a predicted vertex;

[0022] rw is the width of the target detection frame, rh is the height of the target detection frame, xmin is the minimum x-coordinate value of each vertex of the target detection frame, xmax is the maximum x-coordinate value of each vertex of the target detection frame, ymin is the minimum y-coordinate value of each vertex of the target detection frame, and ymax is the maximum y-coordinate value of each vertex of the target detection frame;

[0023] is the x-axis offset variable value corresponding to the vertex (xmin, ymin), The y-axis offset variable value corresponding to the vertex (xmin, ymin);

[0024] is the x-axis offset variable value corresponding to the vertex (xmax, ymin), The y-axis offset variable value corresponding to the vertex (xmax, ymin);

[0025] is the x-axis offset variable value corresponding to the vertex (xmax, ymax), The y-axis offset variable value corresponding to the vertex (xmax, ymax);

[0026] is the x-axis offset variable value corresponding to the vertex (xmin, ymax), It is the y-axis offset variable value corresponding to the vertex (xmin, ymax).

[0027] Furthermore, the image to be detected is processed using a pre-trained neural network model to obtain a target detection frame containing the quadrilateral object and offset variable values ​​corresponding to the four vertices of the target detection frame, including: inputting the image to be detected into the neural network model to obtain candidate detection frames of the area of ​​interest in the image to be detected and the category scores and offset variable values ​​corresponding to each candidate detection frame; determining the candidate detection frame whose category score is higher than a preset score threshold; the candidate detection frame whose category score is higher than the preset score threshold is the target detection frame.

[0028] It should be understood that in the actual detection process, the neural network model may obtain multiple candidate detection frames, so it is necessary to determine the actual target detection frame from these multiple candidate detection frames, so as to further implement the solution of the embodiment of the present application. Based on the neural network classification idea, the neural network model outputs the category score corresponding to each candidate detection frame, so as to effectively realize the determination of the target detection frame.

[0029] Furthermore, before determining the candidate detection frame having a category score higher than a preset score threshold, the method further includes: performing redundancy removal processing on the candidate detection frame.

[0030] It should be understood that in actual applications, the number of candidate detection frames obtained by the neural network model is usually large. In order to improve processing efficiency, the candidate detection frames can be de-redundanted first, thereby reducing the number of candidate detection frames and improving the efficiency of determining the target detection frame.

[0031] An embodiment of the present application provides a model training method, comprising: obtaining a training image set; each training image in the training image set contains a quadrilateral object to be detected and is marked with accurate vertices of the quadrilateral object; using the training image set to train a neural network model to be trained, and obtaining a target detection frame containing the quadrilateral object in each training image, and offset variable values ​​corresponding to four vertices of the target detection frame; offsetting each vertex of the target detection frame according to the offset variable value corresponding to each vertex, and obtaining a predicted vertex of the quadrilateral object; calculating a regression loss value between the predicted vertex and the accurate vertex of the quadrilateral object; determining whether the neural network model has been trained according to the regression loss value; if the training has not been completed, updating the parameters of the neural network model and retraining until the training of the neural network model is determined to be completed according to the regression loss value.

[0032] The neural network model trained in the above manner predicts the vertices of a quadrilateral object based on the four vertices of the target detection box to predict the offset of the vertices of the quadrilateral object. Therefore, the predicted offset is smaller than that of related technologies and is easier to converge during training. When using the neural network model to detect quadrilateral objects, it also outputs the vertices of the target detection box and the offset variable values ​​corresponding to each vertex. Therefore, during detection, the vertices of the quadrilateral object are predicted based on the four vertices of the target detection box, so that even if the quadrilateral object is tilted more than 45 degrees or the tilt angle of the quadrilateral object is particularly small, the detection effect can remain accurate and stable.

[0033] Furthermore, each training image in the training image set is annotated with the category to which the quadrilateral object belongs; after obtaining the predicted vertices of the quadrilateral object, the neural network model will also output the category scores of the quadrilateral box formed by the predicted vertices belonging to each classification category; the method also includes: based on the category scores of the quadrilateral box belonging to each classification category, and the category to which the quadrilateral object corresponding to the quadrilateral box belongs, calculating the classification loss value of the neural network model; determining whether the neural network model has been trained according to the regression loss value, including: determining the total loss value of the neural network model according to the regression loss value and the classification loss value corresponding to the quadrilateral object; when the total loss value converges, determining that the neural network model has been trained; otherwise, determining that the neural network model has not been trained.

[0034] An embodiment of the present application also provides a quadrilateral object detection device, including: a first acquisition module and a processing module; the first acquisition module is used to acquire an image to be detected; the image to be detected contains a quadrilateral object to be detected; the processing module is used to process the image to be detected using a pre-trained neural network model to obtain a target detection frame containing the quadrilateral object, and offset variable values ​​corresponding to four vertices of the target detection frame; the processing module is also used to offset each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object.

[0035] The embodiment of the present application also provides a model training device, including: a second acquisition module and a training module; the second acquisition module is used to acquire a training image set; each training image in the training image set contains a quadrilateral object to be detected and is marked with the accurate vertices of the quadrilateral object; the training module is used to use the training image set to train the neural network model to be trained, and obtain a target detection frame containing the quadrilateral object in each training image, and offset variable values ​​corresponding to the four vertices of the target detection frame; the training module is also used to offset each vertex of the target detection frame according to the offset variable value corresponding to each vertex, to obtain the predicted vertex of the quadrilateral object, and calculate the regression loss value between the predicted vertex and the accurate vertex of the quadrilateral object, and determine whether the neural network model is trained according to the regression loss value; if the training is not completed, the parameters of the neural network model are updated and re-trained until the training of the neural network model is determined to be completed according to the regression loss value.

[0036] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a communication bus; the communication bus is used to realize connection and communication between the processor and the memory; the processor is used to execute one or more programs stored in the memory to realize any of the above-mentioned quadrilateral object detection methods, or to realize any of the above-mentioned model training methods.

[0037] In an embodiment of the present application, a readable storage medium is also provided, which stores one or more programs. The one or more programs can be executed by one or more processors to implement any of the above-mentioned quadrilateral object detection methods, or to implement any of the above-mentioned model training methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a flow chart of a quadrilateral object detection method provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of a quadrilateral object detection device provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0046] Embodiment 1:

[0047] The embodiments of the present application provide a model training method and a quadrilateral object detection method. The neural network model trained by the model training method can be used to implement position detection of quadrilateral objects in the quadrilateral object detection method.

[0048] See also Figure 1 As shown, the model training method includes:

[0049] S101: Obtain a training image set.

[0050] In the embodiment of the present application, engineers can obtain training images containing quadrilateral objects to be detected through various methods (such as on-site shooting, network collection, etc.) and summarize them into a training image set.

[0051] In an embodiment of the present application, in order to facilitate training, pre-processing operations such as grayscale conversion and resolution adjustment may be performed on the collected images.

[0052] It should be noted that in the embodiment of the present application, for each training image in the training image set, the engineer needs to mark the exact vertices of the quadrilateral object to be detected in each training image to facilitate the loss calculation.

[0053] S102: Using the training image set to train the neural network model to be trained, obtain the target detection frame containing the quadrilateral object in each training image, and the offset variable values ​​corresponding to the four vertices of the target detection frame.

[0054] In an embodiment of the present application, the last layer of the neural network model can be configured to output the coordinates of vertices on at least two diagonal corners of the target detection frame (since the detection frame is a horizontal rectangle, the coordinates of the two diagonal vertices are determined, then the coordinates of the four vertices are actually all known), and the last layer of the neural network model can be configured to also output 8 offset variable values, which respectively represent the offsets of the x-axis and y-axis relative to the four vertices of the target detection frame.

[0055] It should be understood that, in the embodiment of the present application, each vertex of the target detection frame corresponds to two offset variable values. Of the two corresponding offset variable values, one is the x-axis offset variable value corresponding to the x-coordinate value of the target detection frame, and the other is the y-axis offset variable value corresponding to the y-coordinate value of the target detection frame, so that the predicted coordinates of the four vertices of the quadrilateral can be generated based on the four vertices of the target detection frame.

[0056] It should be noted that in the embodiments of the present application, the neural network model needs to adopt a neural network model that can realize the acquisition of the detection frame of the object, for example, the Faster RCNN model, the efficientdet model, etc.

[0057] S103: offset each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object.

[0058] In an embodiment of the present application, the configured offset variable can be the ratio of the vertices of the quadrilateral object relative to the four vertices of the target detection frame, and the offset in the x-axis direction and the y-axis direction relative to the width and height of the target detection frame (which can be positive or negative).

[0059] At this time, there is:

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068] In the above formula: x0 and y0 constitute the coordinates of a predicted vertex of the quadrilateral object, x1 and y1 constitute the coordinate value of a predicted vertex of the quadrilateral object, x2 and y2 constitute the coordinate value of a predicted vertex of the quadrilateral object, and x3 and y3 constitute the coordinate value of a predicted vertex of the quadrilateral object.

[0069] In addition, in the above formula: rw is the width of the target detection box, rh is the height of the target detection box, xmin is the minimum x coordinate value of each vertex of the target detection box, xmax is the maximum x coordinate value of each vertex of the target detection box, ymin is the minimum y coordinate value of each vertex of the target detection box, and ymax is the maximum y coordinate value of each vertex of the target detection box;

[0070] is the x-axis offset variable value corresponding to the vertex (xmin, ymin), The y-axis offset variable value corresponding to the vertex (xmin, ymin);

[0071] is the x-axis offset variable value corresponding to the vertex (xmax, ymin), The y-axis offset variable value corresponding to the vertex (xmax, ymin);

[0072] is the x-axis offset variable value corresponding to the vertex (xmax, ymax), The y-axis offset variable value corresponding to the vertex (xmax, ymax);

[0073] is the x-axis offset variable value corresponding to the vertex (xmin, ymax), It is the y-axis offset variable value corresponding to the vertex (xmin, ymax).

[0074] It should be understood that in the embodiment of the present application, in addition to setting the offset variable to be the ratio of the offset of the vertices of the quadrilateral object relative to the four vertices of the target detection frame in the x-axis direction and the y-axis direction to the width and height of the target detection frame, the offset variable can also be directly set to be the offset of the vertices of the quadrilateral object relative to the four vertices of the target detection frame in the x-axis direction and the y-axis direction (which can be positive or negative). In this case, the calculation formula for the predicted vertices of the quadrilateral object is to directly add the x-coordinate or y-coordinate of the target detection frame to the corresponding x-axis offset or y-axis offset.

[0075] S104: Calculate the regression loss value between the predicted vertices and the accurate vertices of the quadrilateral object.

[0076] In the embodiment of the present application, the regression loss function for calculating the regression loss value can be set by the engineer according to actual needs. As long as the deviation agreement between the predicted vertex and the accurate vertex is met, the larger the regression loss value is, the better.

[0077] It should be understood that in actual application, the inventors found that the deviation of the vertices of a large number of quadrilateral objects is close to 0, while the deviation of a few quadrilateral boxes (i.e., boxes obtained based on the four predicted vertices of the quadrilateral object) is large. Therefore, in a feasible implementation of the embodiment of the present application, the classification focal loss function can be referred to to increase the penalty for the predicted quadrilateral boxes with large deviations. The loss function is as follows:

[0078]

[0079] In the formula, X i ,Y i is the x-coordinate value and y-coordinate value of the exact vertex of the quadrilateral object, L reg is the regression loss value.

[0080] S105: Determine whether the neural network model has been trained based on the regression loss value. If not, go to step S106; if trained, end.

[0081] In a feasible example of an embodiment of the present application, whether the neural network model is trained can be determined by judging whether the regression loss value converges.

[0082] It should be understood that in an image, there may be more than one quadrilateral object, so there may be more than one corresponding quadrilateral box. Therefore, the regression loss values ​​corresponding to all quadrilateral boxes can be combined to determine whether the neural network model has been trained.

[0083] For example, the formula The regression loss values ​​corresponding to all quadrilateral boxes are combined to obtain the average regression loss value L, and then determine whether L converges. If it converges, the training is completed; otherwise, the training is not completed.

[0084] It should be noted that, in the above formula, A is the number of all quadrilateral frames, is the regression loss value of the a-th quadrilateral box, To find the sum of the regression loss values ​​of all quadrilateral boxes.

[0085] In an embodiment of the present application, the neural network model may adopt a classification model, so that the target detection frame may be determined by the category score of each candidate detection frame output by the neural network model.

[0086] It should be understood that, for a classification model, it will output an N-dimensional vector (assuming there are N categories in total), and each value in the N-dimensional vector corresponds to the category score of a category.

[0087] In practical applications, the quadrilateral objects to be located are usually also objects that need to be classified and identified. Based on this, by comparing the category scores of each candidate detection frame with the preset score threshold, the target detection frame can be effectively screened out. The category score of the target detection frame is the category score of the quadrilateral frame corresponding to the target detection frame that is subsequently determined.

[0088] It should be understood that in actual application, the number of candidate detection frames obtained by the neural network model is usually large. In order to improve processing efficiency, in the embodiment of the present application, the candidate detection frames can be first de-redundant (for example, non-maximum suppression calculation is used for all candidate detection frames to achieve de-redundant processing), thereby reducing the number of candidate detection frames and improving the efficiency of determining the target detection frame.

[0089] In order to ensure the classification and recognition reliability of the neural network model, it is necessary to calculate the classification loss value of the neural network model.

[0090] The classification loss function used to calculate the classification loss value can be set by the engineer according to actual needs. For example, the classification loss function for each quadrilateral box or target detection box can be:

[0091]

[0092] In the above formula, K is the number of categories (the reason why the total number of categories is K+1 is that the image that is not in the K category is also considered as a category), r is a power hyperparameter (set by engineers according to actual needs, or obtained through a large amount of experimental data, and can be 2 in the embodiment of the present application), and y represents whether the quadrilateral object corresponding to the quadrilateral box or the target detection box is of this category, which is 1 if it is and 0 if it is not.

[0093] In an embodiment of the present application, the classification loss value and the regression loss value can be combined to determine whether the neural network model has been trained.

[0094] Exemplarily, the total loss value of the neural network model can be determined according to the regression loss value and the classification loss value corresponding to the quadrilateral object, for example, the total loss value can be determined according to the following formula:

[0095]

[0096] In the formula, A is the number of all quadrilateral boxes, is the regression loss value of the a-th quadrilateral box, is the classification loss value of the a-th quadrilateral box.

[0097] When the total loss value converges, it is determined that the neural network model has been trained; otherwise, it is determined that the neural network model has not been trained.

[0098] S106: Update the parameters of the neural network model and go to S102 for retraining.

[0099] After the neural network model is trained, the neural network model can be used to detect the position of quadrilateral objects.

[0100] See also Figure 2 As shown, the quadrilateral object detection method includes:

[0101] S201: Acquire an image to be detected.

[0102] The image to be detected contains a quadrilateral object to be detected.

[0103] S202: Use the trained neural network model to process the image to be detected to obtain a target detection frame containing a quadrilateral object and offset variable values ​​corresponding to four vertices of the target detection frame.

[0104] S203: offset each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object.

[0105] In the embodiment of the present application, the vertex has an x-coordinate value and a y-coordinate value, and the relevant introduction of the offset variable value can be found in the introduction of the aforementioned training process. Under different definitions of the offset variable value, the calculation corresponding to the predicted vertex is different.

[0106] Exemplarily, when the configured offset variable is the ratio of the vertices of the quadrilateral object relative to the four vertices of the target detection frame, and the offset in the x-axis direction and the y-axis direction relative to the width and height of the target detection frame, the width and height of the target detection frame can be determined, and the x-coordinate predicted values ​​of each predicted vertex of the quadrilateral object can be determined according to the width, the x-coordinate values ​​of each vertex of the target detection frame, and the corresponding x-axis offset variable values; the y-coordinate predicted values ​​of each predicted vertex of the quadrilateral object can be determined according to the height, the y-coordinate values ​​of each vertex of the target detection frame, and the corresponding y-axis offset variable values.

[0107] At this time, there is:

[0108]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116] In the above formula: x0 and y0 constitute the coordinates of a predicted vertex of the quadrilateral object, x1 and y1 constitute the coordinate value of a predicted vertex of the quadrilateral object, x2 and y2 constitute the coordinate value of a predicted vertex of the quadrilateral object, and x3 and y3 constitute the coordinate value of a predicted vertex of the quadrilateral object.

[0117] In addition, in the above formula: rw is the width of the target detection box, rh is the height of the target detection box, xmin is the minimum x coordinate value of each vertex of the target detection box, xmax is the maximum x coordinate value of each vertex of the target detection box, ymin is the minimum y coordinate value of each vertex of the target detection box, and ymax is the maximum y coordinate value of each vertex of the target detection box;

[0118] is the x-axis offset variable value corresponding to the vertex (xmin, ymin), The y-axis offset variable value corresponding to the vertex (xmin, ymin);

[0119] is the x-axis offset variable value corresponding to the vertex (xmax, ymin), The y-axis offset variable value corresponding to the vertex (xmax, ymin);

[0120] is the x-axis offset variable value corresponding to the vertex (xmax, ymax), The y-axis offset variable value corresponding to the vertex (xmax, ymax);

[0121] is the x-axis offset variable value corresponding to the vertex (xmin, ymax), It is the y-axis offset variable value corresponding to the vertex (xmin, ymax).

[0122] It should be noted that in the actual detection process, the neural network model may obtain multiple candidate detection frames, so it is necessary to determine the actual target detection frame from these multiple candidate detection frames in order to further implement the solution of the embodiment of the present application. Based on the neural network classification idea, the neural network model outputs the category score corresponding to each candidate detection frame, which can effectively realize the determination of the target detection frame.

[0123] To this end, in the embodiment of the present application, the neural network model can adopt a classification model. When processing the image to be detected, the neural network model also outputs an N-dimensional vector of each candidate detection frame, and each value in the N-dimensional vector represents the score of the object in the candidate detection frame belonging to a certain category.

[0124] The category scores corresponding to the candidate detection frames can be compared with the preset score thresholds respectively (or the highest category score among the category scores corresponding to the candidate detection frames can be directly compared with the preset score thresholds), so as to determine the candidate detection frame whose category score is higher than the preset score threshold. The determined candidate detection frame whose category score is higher than the preset score threshold is the target detection frame.

[0125] It should be understood that in actual application, the number of candidate detection frames obtained by the neural network model is usually large. In order to improve processing efficiency, in an embodiment of the present application, the candidate detection frames can be first de-redundanted, thereby reducing the number of candidate detection frames, and then the target detection frame can be determined, thereby improving the efficiency of determining the target detection frame.

[0126] The quadrilateral object detection method and model training method provided in the embodiments of the present application directly outputs a target detection frame containing a quadrilateral object and the offset variable values ​​corresponding to the four vertices of the target detection frame through a neural network model, and then offsets each vertex of the target detection frame according to the offset variable values ​​corresponding to each vertex to obtain the predicted vertices of the quadrilateral object. Compared with the related art, the predicted vertices of the quadrilateral object obtained in this way are not restricted by the edges of the target detection frame, and the vertices of the quadrilateral object are not limited to the edges of the target detection frame. That is, the offset of the vertices of the quadrilateral object is not predicted based on the edges of the target detection frame, but based on the four vertices of the target detection frame, so that even when the quadrilateral object is tilted more than 45 degrees or the tilt angle of the quadrilateral object is particularly small, the detection effect can remain accurate and stable.

[0127] Embodiment 2:

[0128] This embodiment, based on the first embodiment, takes a specific training and detection process as an example to illustrate the solution of the present application.

[0129] See also Figure 3 As shown, Figure 3 A specific structure of a neural network model is provided in an embodiment of the present application.

[0130] Figure 3 In the example, Faster RCNN is used as the detection framework, and the coordinates of the upper left corner vertex and the lower right corner vertex of the target detection box are output in the last layer to obtain four coordinate values ​​xmin, xmax, ymin, and ymax.

[0131] Figure 3 In the last layer, 8 offset variables are added The values ​​of the eight offset variables represent the x-axis and y-axis offsets of the four vertices of the target detection frame, thereby generating the coordinates of the four vertices of the quadrilateral object. The calculation formula is:

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] In the formula, rw=xmax-xmin, rh=ymax-ymin.

[0141] Next, the actual values ​​of the coordinates of the four vertices of the quadrilateral object are used to calculate the deviation from the predicted values ​​and the network loss.

[0142] Network loss function:

[0143] The loss function of each anchor (quadrilateral box) category is as follows, where K is the number of categories, r is the power hyperparameter, which is 2 in this project, and y represents whether it is this category, which is 1 if it is, and 0 if it is not:

[0144]

[0145] The regression loss function of each anchor is as follows, where X i , Y i The real coordinate value corresponding to the quadrilateral object:

[0146]

[0147] For all anchors, the overall network loss function is as follows:

[0148]

[0149] In the formula, A is the number of all anchors, is the regression loss value of the a-th anchor, is the classification loss value of the a-th anchor.

[0150] During the detection phase, the processing process is as follows:

[0151] 1. Take the image to be detected as the input of the neural network model, forward pass it, and calculate the category of each ROI (region of interest) of the RPN (Region Proposal Network) in Faster RCNN, the candidate detection box, the coordinates of the upper left corner vertex and the lower right corner fixed point of the candidate detection box, and the corresponding 8 offset variable values.

[0152] 2. For all ROI calculation results, use NMS (non maximum suppression) to calculate and initially obtain sparse candidate detection frames. Then, according to the category scores of the retained candidate detection frames, filter through the threshold to obtain the target detection frame.

[0153] 3. For each target detection frame, according to the vertex coordinates of the target detection frame and the corresponding 8 offset variable values, the four vertex coordinates of the quadrilateral object are calculated according to the calculation formula in the previous text.

[0154] Embodiment three:

[0155] Based on the same inventive concept, the present application also provides a quadrilateral object detection device and a model training device applied to an electronic device. Figure 4 and Figure 5 As shown, Figure 4 The quadrilateral object detection device 100 corresponding to the quadrilateral object detection method shown in the first embodiment is shown. Figure 5 The model training device 200 corresponding to the model training method shown in the first embodiment is shown. It should be understood that the specific functions of the quadrilateral object detection device 100 and the model training device 200 can be referred to the description above. To avoid repetition, the detailed description is appropriately omitted here. The quadrilateral object detection device 100 and the model training device 200 include at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system of the quadrilateral object detection device 100 and the model training device 200. Specifically:

[0156] See also Figure 4 As shown, the quadrilateral object detection device 100 includes: a first acquisition module 101 and a processing module 102. Among them:

[0157] The first acquisition module 101 is used to acquire an image to be detected; the image to be detected contains a quadrilateral object to be detected;

[0158] The processing module 102 is used to process the image to be detected using a pre-trained neural network model to obtain a target detection frame containing the quadrilateral object and offset variable values ​​corresponding to four vertices of the target detection frame;

[0159] The processing module 102 is further configured to offset each vertex of the target detection frame according to an offset variable value corresponding to each vertex to obtain a predicted vertex of the quadrilateral object.

[0160] In an embodiment of the present application, the vertex has an x-coordinate value and a y-coordinate value; the offset variable value corresponding to the vertex includes: an x-axis offset variable value corresponding to the x-coordinate value of the vertex, and a y-axis offset variable value corresponding to the y-coordinate value of the vertex; the processing module 102 is specifically used to determine the width and height of the target detection frame; determine the x-coordinate prediction value of each predicted vertex of the quadrilateral object according to the width, the x-coordinate value of each vertex of the target detection frame and the corresponding x-axis offset variable value; determine the y-coordinate prediction value of each predicted vertex of the quadrilateral object according to the height, the y-coordinate value of each vertex of the target detection frame and the corresponding y-axis offset variable value.

[0161] In a feasible implementation of the embodiment of the present application, the processing module 102 is specifically used to determine the predicted x-coordinate value and the predicted y-coordinate value of each predicted vertex of the quadrilateral object according to the following formula:

[0162]

[0163]

[0164]

[0165]

[0166]

[0167]

[0168]

[0169]

[0170] in:

[0171] x0 and y0 constitute the coordinates of a predicted vertex, x1 and y1 constitute the coordinate value of a predicted vertex, x2 and y2 constitute the coordinate value of a predicted vertex, and x3 and y3 constitute the coordinate value of a predicted vertex;

[0172] rw is the width of the target detection frame, rh is the height of the target detection frame, xmin is the minimum x-coordinate value of each vertex of the target detection frame, xmax is the maximum x-coordinate value of each vertex of the target detection frame, ymin is the minimum y-coordinate value of each vertex of the target detection frame, and ymax is the maximum y-coordinate value of each vertex of the target detection frame;

[0173] is the x-axis offset variable value corresponding to the vertex (xmin, ymin), The y-axis offset variable value corresponding to the vertex (xmin, ymin);

[0174] is the x-axis offset variable value corresponding to the vertex (xmax, ymin), The y-axis offset variable value corresponding to the vertex (xmax, ymin);

[0175] is the x-axis offset variable value corresponding to the vertex (xmax, ymax), The y-axis offset variable value corresponding to the vertex (xmax, ymax);

[0176] is the x-axis offset variable value corresponding to the vertex (xmin, ymax), It is the y-axis offset variable value corresponding to the vertex (xmin, ymax).

[0177] In a feasible implementation manner of the embodiment of the present application, the processing module 102 is specifically used to input the image to be detected into the neural network model, obtain the candidate detection frame of the area of ​​interest in the image to be detected and the category score and offset variable value corresponding to each candidate detection frame; determine the candidate detection frame whose category score is higher than a preset score threshold; the candidate detection frame whose category score is higher than the preset score threshold is the target detection frame.

[0178] In the above feasible implementation manner, the processing module 102 is further configured to perform redundancy removal processing on the candidate detection frame before determining the candidate detection frame whose category score is higher than a preset score threshold.

[0179] See also Figure 5 As shown, the model training device 200 includes: a second acquisition module 201 and a training module 202. Among them:

[0180] The second acquisition module 201 is used to acquire a training image set; each training image in the training image set contains a quadrilateral object to be detected and is marked with accurate vertices of the quadrilateral object;

[0181] The training module 202 is used to train the neural network model to be trained using the training image set to obtain the target detection frame containing the quadrilateral object in each training image and the offset variable values ​​corresponding to the four vertices of the target detection frame;

[0182] The training module 202 is also used to offset each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object, and calculate the regression loss value between the predicted vertex and the accurate vertex of the quadrilateral object, and determine whether the neural network model is trained according to the regression loss value; if the training is not completed, update the parameters of the neural network model and re-train until the training of the neural network model is determined to be completed according to the regression loss value.

[0183] In an embodiment of the present application, each training image in the training image set is annotated with the category to which the quadrilateral object belongs;

[0184] The training module 202 is further configured to output the category scores of each classification category of the quadrilateral box formed by the predicted vertices after obtaining the predicted vertices of the quadrilateral object;

[0185] The training module 202 is further used to calculate the classification loss value of the neural network model based on the category scores of each classification category to which the quadrilateral box belongs and the category to which the quadrilateral object corresponding to the quadrilateral box belongs.

[0186] And determining whether the neural network model has been trained based on the regression loss value includes: determining the total loss value of the neural network model based on the regression loss value and the classification loss value corresponding to the quadrilateral object; when the total loss value converges, determining that the neural network model has been trained; otherwise, determining that the neural network model has not been trained.

[0187] It should be understood that, for the sake of brevity, some of the contents described in the first embodiment will not be repeated in this embodiment.

[0188] Embodiment 4:

[0189] This embodiment provides an electronic device, which can be seen in Figure 6 As shown, it includes a processor 601, a memory 602, and a communication bus 603. Among them:

[0190] The communication bus 603 is used to realize the connection and communication between the processor 601 and the memory 6023 .

[0191] The processor 601 is used to execute one or more programs stored in the memory 602 to implement the quadrilateral object detection method or model training method in the above-mentioned embodiment 1 / 2.

[0192] Understandably, Figure 6 The structure shown is for illustration only. The electronic device may also include Figure 6 More or fewer components as shown, or with Figure 6The different configurations shown may also include input / output ports, display screens, and other components.

[0193] This embodiment also provides a readable storage medium, such as a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash disk, an SD (Secure Digital Memory Card) card, an MMC (Multimedia Card) card, etc., in which one or more programs for implementing the above steps are stored, and the one or more programs can be executed by one or more processors to implement the quadrilateral object detection method or model training method in the above embodiment 1 / 2. No further details will be given here.

[0194] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0195] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0197] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0198] As used herein, a plurality refers to two or more than two.

[0199] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: include: Get a training image set; Each training image in the training image set contains a quadrilateral object to be detected and is marked with accurate vertices of the quadrilateral object; The neural network model to be trained is trained using the training image set to obtain a target detection frame containing the quadrilateral object in each training image, and offset variable values ​​corresponding to four vertices of the target detection frame; Offsetting each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object; Calculating a regression loss value between the predicted vertices and the accurate vertices of the quadrilateral object; Determining whether the neural network model has been trained according to the regression loss value; If the training is not completed, the parameters of the neural network model are updated and the training is performed again until the training of the neural network model is determined to be completed according to the regression loss value; Wherein: calculating the regression loss value between the predicted vertex and the accurate vertex of the quadrilateral object includes: According to the formula Calculate the regression loss value between the predicted vertices and the accurate vertices of the quadrilateral object; where: A is the number of all quadrilateral boxes, is the regression loss value L of the a-th quadrilateral box reg , To find the sum of the regression loss values ​​of all quadrilateral boxes; where: the regression loss value L of a single quadrilateral box reg for: Among them, X0, X1, X2, X3, Y0, Y1, Y2, and Y3 are the x-coordinate values ​​and y-coordinate values ​​of the accurate vertices of the quadrilateral object, and x0, x1, x2, x3, y0, y1, y2, and y3 are the x-coordinate values ​​and y-coordinate values ​​of the predicted vertices of the quadrilateral object.

2. The model training method according to claim 1, characterized in that: Each training image in the training image set is annotated with the category to which the quadrilateral object belongs; After obtaining the predicted vertices of the quadrilateral object, the neural network model also outputs the category scores of each classification category of the quadrilateral box formed by the predicted vertices; The method further comprises: Calculating the classification loss value of the neural network model based on the category scores of each classification category to which the quadrilateral box belongs and the category to which the quadrilateral object corresponding to the quadrilateral box belongs; Determining whether the neural network model is trained according to the regression loss value includes: Determine the total loss value of the neural network model according to the regression loss value and the classification loss value corresponding to the quadrilateral object; When the total loss value converges, it is determined that the neural network model has been trained; otherwise, it is determined that the neural network model has not been trained.

3. A method for detecting a quadrilateral object, characterized in that: include: Acquire an image to be detected; the image to be detected contains a quadrilateral object to be detected; The image to be detected is processed using a pre-trained neural network model to obtain a target detection frame containing the quadrilateral object and offset variable values ​​corresponding to four vertices of the target detection frame; the neural network model is trained by the model training method according to claim 1 or 2; Each vertex of the target detection frame is offset according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object.

4. The method for detecting a quadrilateral object according to claim 3, wherein: The vertex has an x-coordinate value and a y-coordinate value; the offset variable value corresponding to the vertex includes: an x-axis offset variable value corresponding to the x-coordinate value of the vertex, and a y-axis offset variable value corresponding to the y-coordinate value of the vertex; Offsetting each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object, including: Determine the width and height of the target detection frame; Determine the predicted x-coordinate value of each predicted vertex of the quadrilateral object according to the width, the x-coordinate value of each vertex of the target detection frame, and the corresponding x-axis offset variable value; According to the height, the y coordinate values ​​of each vertex of the target detection frame and the corresponding y-axis offset variable value, the y coordinate predicted values ​​of each predicted vertex of the quadrilateral object are determined.

5. The method for detecting a quadrilateral object according to claim 4, wherein: Determining the predicted x-coordinate value of each predicted vertex of the quadrilateral object according to the width, the x-coordinate value of each vertex of the target detection frame, and the corresponding x-axis offset variable value; and determining the predicted y-coordinate value of each predicted vertex of the quadrilateral object according to the height, the y-coordinate value of each vertex of the target detection frame, and the corresponding y-axis offset variable value, including: The predicted x-coordinate value and the predicted y-coordinate value of each predicted vertex of the quadrilateral object are determined according to the following formula: in: x0 and y0 constitute the coordinates of a predicted vertex, x1 and y1 constitute the coordinate value of a predicted vertex, x2 and y2 constitute the coordinate value of a predicted vertex, and x3 and y3 constitute the coordinate value of a predicted vertex; rw is the width of the target detection frame, rh is the height of the target detection frame, xmin is the minimum x-coordinate value of each vertex of the target detection frame, xmax is the maximum x-coordinate value of each vertex of the target detection frame, ymin is the minimum y-coordinate value of each vertex of the target detection frame, and ymax is the maximum y-coordinate value of each vertex of the target detection frame; is the x-axis offset variable value corresponding to the vertex (ymin, ymin), The y-axis offset variable value corresponding to the vertex (xmin, ymin); is the x-axis offset variable value corresponding to the vertex (xmax, ymin), The y-axis offset variable value corresponding to the vertex (xmax, ymin); is the x-axis offset variable value corresponding to the vertex (xmax, ymax), The y-axis offset variable value corresponding to the vertex (xmax, ymax); is the x-axis offset variable value corresponding to the vertex (xmin, ymax), It is the y-axis offset variable value corresponding to the vertex (xmin, ymax).

6. The method for detecting a quadrilateral object according to any one of claims 3 to 5, characterized in that: The image to be detected is processed using a pre-trained neural network model to obtain a target detection frame containing the quadrilateral object and offset variable values ​​corresponding to four vertices of the target detection frame, including: Inputting the image to be detected into the neural network model, obtaining candidate detection frames of the region of interest in the image to be detected and the category score and offset variable value corresponding to each candidate detection frame; The candidate detection frame whose category score is higher than a preset score threshold is determined; the candidate detection frame whose category score is higher than the preset score threshold is the target detection frame.

7. The method for detecting a quadrilateral object according to claim 6, wherein: Before determining the candidate detection frame whose category score is higher than a preset score threshold, the method further includes: De-redundancy processing is performed on the candidate detection frame.

8. A quadrilateral object detection device, characterized in that: include: A first acquisition module and a processing module; The first acquisition module is used to acquire an image to be detected; the image to be detected contains a quadrilateral object to be detected; The processing module is used to process the image to be detected using a pre-trained neural network model to obtain a target detection frame containing the quadrilateral object and offset variable values ​​corresponding to four vertices of the target detection frame; the neural network model is trained according to the model training method according to claim 1 or 2; The processing module is further used to offset each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object.

9. A model training device, characterized in that: include: Second acquisition module and training module; The second acquisition module is used to acquire a training image set; Each training image in the training image set contains a quadrilateral object to be detected and is marked with accurate vertices of the quadrilateral object; The training module is used to train the neural network model to be trained using the training image set to obtain a target detection frame containing the quadrilateral object in each training image, and offset variable values ​​corresponding to four vertices of the target detection frame; The training module is further used to offset each vertex of the target detection frame according to the offset variable value corresponding to each vertex to obtain the predicted vertex of the quadrilateral object, and calculate the regression loss value between the predicted vertex and the accurate vertex of the quadrilateral object, and determine whether the neural network model is trained according to the regression loss value; If the training is not completed, the parameters of the neural network model are updated and the training is performed again until the training of the neural network model is determined to be completed according to the regression loss value; Wherein: calculating the regression loss value between the predicted vertex and the accurate vertex of the quadrilateral object includes: According to the formula Calculate the regression loss value between the predicted vertices and the accurate vertices of the quadrilateral object; where: A is the number of all quadrilateral boxes, is the regression loss value L of the a-th quadrilateral box reg , To find the sum of the regression loss values ​​of all quadrilateral boxes; where: the regression loss value L of a single quadrilateral box reg for: Among them, X0, X1, X2, X3, Y0, Y1, Y2, and Y3 are the x-coordinate values ​​and y-coordinate values ​​of the accurate vertices of the quadrilateral object, and x0, x1, x2, x3, y0, y1, y2, and y3 are the x-coordinate values ​​and y-coordinate values ​​of the predicted vertices of the quadrilateral object.

10. An electronic device, characterized in that: include: Processor, memory and communication bus; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to execute one or more programs stored in the memory to implement the quadrilateral object detection method as described in any one of claims 3 to 7, or to implement the model training method as described in claim 1 or 2.

11. A readable storage medium, characterized in that: The readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the quadrilateral object detection method as described in any one of claims 3 to 7, or to implement the model training method as described in claim 1 or 2.