Detection method, device, and computer-readable storage medium

Abstract

The present invention provides a detection method, comprising the steps of: classifying and identifying first-class boxes and second-class boxes in a neural network model and separately limiting the positions of the first-class boxes and the second-class boxes to obtain the positions of the first-class boxes and the second-class boxes; extracting comprehensive semantic information from the first-class boxes and the second-class boxes; determining the inclusion relationship between the first-class boxes and the second-class boxes, and when it is determined that the inclusion relationship between the first-class boxes and the second-class boxes is correct, performing data correction to reduce the weight when the relative positions are correct, thereby constraining the relative positions of the first-class boxes and the second-class boxes so that the first-class boxes are located within the second-class boxes. This solves the problem of the lack of relative position constraints between the second-class boxes and the first-class boxes, and improves the accuracy of the relative positions of the second-class boxes and the first-class boxes. The present invention also provides a device and a computer-readable storage medium.

Description

Technical Field

[0001] The present invention relates to the field of detection technology, and in particular to a detection method, device and computer-readable storage medium. Background Art

[0002] In recent years, with the continuous development of intelligent recognition technology, intelligent video surveillance has played an increasingly important role in many fields, including security and intelligent transportation. Cameras can continuously record everything within their field of view, but the video they produce is unstructured data, meaning that objects in the video can only be viewed visually and cannot be directly read or identified by computers.

[0003] Pedestrians are a key focus of video surveillance systems. Pedestrian structuring involves extracting pedestrian-related information from video content, converting it into machine-understandable information, and further transforming it into intelligence, thus transforming video data into information and intelligence. Pedestrian-related descriptive information can include pedestrian location, facial location, facial features, gender, age range, and movement trajectory.

[0004] However, existing techniques generally train and deploy two related categories of objects, such as face boxes and human-shaped boxes, separately. Since the two categories are treated as independent categories, their positional information is not integrated during training, resulting in completely independent relative positions of the first and second category boxes after detection. This can lead to problems such as being unable to quickly and effectively determine whether the two detected categories of objects belong to the same target. This can also affect the use of the detection results of the two categories for other applications, resulting in low test accuracy and test errors. Furthermore, existing techniques lack the integration of semantic information of the two related categories during model training, increasing computational overhead and memory usage.

[0005] Therefore, it is necessary to provide a new detection method, device and computer-readable storage medium to solve the above problems existing in the prior art. Summary of the Invention

[0006] The object of the present invention is to provide a detection method, device and computer-readable storage medium, which solve the problem of lack of relative position constraints between the second type frame and the first type frame, and improve the accuracy of the relative position of the second type frame and the first type frame.

[0007] To achieve the above object, the detection method of the present invention comprises the following steps:

[0008] S100: Classifying and identifying first-category boxes and second-category boxes in a neural network model and separately defining the positions of the first-category boxes and the second-category boxes to obtain first-category box positions and second-category box positions; and extracting comprehensive semantic information from the first-category boxes and the second-category boxes to integrate the semantic information of the first-category boxes and the second-category boxes and output the first-category box information and the second-category box information.

[0009] S200: constraining the relative positions of the first-type frame and the second-type frame so that the first-type frame is located within the second-type frame;

[0010] The step S200 specifically includes: determining the inclusion relationship between the first type of frame and the second type of frame, and when it is determined that the inclusion relationship between the first type of frame and the second type of frame is correct, performing data correction to reduce the weight when the relative position is correct.

[0011] The detection method of the present invention has the beneficial effect of: constraining the relative positions of the first-category box position and the second-category box position so that the first-category box is located within the second-category box; step S200 specifically includes: determining the inclusion relationship between the first-category box and the second-category box; when it is determined that the inclusion relationship between the first-category box and the second-category box is correct, correcting the data to reduce the weight when the relative position is correct, thereby solving the problem of the lack of relative position constraints between the second-category box and the first-category box, and improving the accuracy of the relative position of the second-category box and the first-category box; extracting comprehensive semantic information from the first-category box and the second-category box to integrate the semantic information of the first-category box and the second-category box and outputting the first-category box information and the second-category box information, so that the semantic information of the first-category box and the second-category box can be better integrated, thereby improving the detection rate of the first-category box and the second-category box, thereby obtaining higher detection accuracy; and classifying and identifying the first-category box and the second-category box in the neural network model and separately limiting the positions of the first-category box and the second-category box to obtain the first-category box position and the second-category box position, thereby reducing the computational consumption and memory usage during model training of the neural network model.

[0012] Preferably, step S200 further includes determining the inclusion relationship between the first-category frame and the second-category frame, and when it is determined that the inclusion relationship between the first-category frame and the second-category frame is correct, performing data correction to reduce the weight when the relative position is correct includes:

[0013] S201: performing normalization operations on the center point, width, and height of the predicted first-category box and the predicted second-category box, respectively, to obtain the upper-left point coordinates and the lower-right point coordinates of the approximate first-category box and the approximate second-category box, respectively;

[0014] S202: performing horizontal and vertical comparison operations on the obtained upper left point coordinates and lower right point coordinates of the approximate first category frame and the upper left point coordinates and lower right point coordinates of the approximate second category frame respectively to obtain comparison parameters, and performing a sigmoid function operation on the comparison parameters to obtain a sigmoid function result;

[0015] S203: Applying the focal loss function to the sigmoid function results and then performing a summation operation to obtain a regression loss result for the relative positions of the first and second category boxes. This has the beneficial effect of confirming the inclusion relationship between the first and second category boxes through the sigmoid function operation, and then using the focal loss function to reduce the weight of the correct relative positions, thereby allowing the model to focus more on correcting the boxes with incorrect relative positions during training.

[0016] Preferably, after step S203 is executed, the following steps are further performed:

[0017] S204: Repeat steps S201 to S203 for all candidate boxes in all grids of the neural network model that have both the first and second class boxes, and sum the obtained regression loss results for the relative positions of all first and second class boxes to obtain a total regression loss result for the relative positions of the first and second class boxes. This has the beneficial effect of preventing missed detections and improving the accuracy of the relative positions of the second and first class boxes.

[0018] Preferably, after step S204 is executed, the following steps are further performed:

[0019] S205: Multiplying the total regression loss result of the relative position of the first-class frame and the second-class frame by the weight coefficient of the relative position loss of the first-class frame and the second-class frame to increase the relative position loss of the first-class frame and the second-class frame. This has the beneficial effect of making the calculation more accurate by increasing the relative position loss of the first-class frame and the second-class frame, thereby improving the accuracy of the relative position of the first-class frame and the second-class frame.

[0020] Preferably, the contrast parameters in step S202 include a first horizontal contrast parameter, a second horizontal contrast parameter, a first vertical contrast parameter, and a second vertical contrast parameter;

[0021] The first horizontal contrast parameter is obtained by subtracting the x-coordinate value of the upper left point of the approximate second type frame from the x-coordinate value of the upper left point of the approximate first type frame;

[0022] The first vertical contrast parameter is obtained by subtracting the y coordinate value of the upper left point of the approximate second type frame from the y coordinate value of the upper left point of the approximate first type frame;

[0023] The second horizontal contrast parameter is obtained by subtracting the x-coordinate value of the lower right point of the approximate second type frame from the x-coordinate value of the lower right point of the approximate second type frame;

[0024] The second vertical contrast parameter is obtained by subtracting the y-coordinate value of the lower right point of the approximate second-type frame from the y-coordinate value of the lower right point of the approximate first-type frame. This advantageously adds relative position constraints to the second-type frame and the first-type frame, thereby improving the accuracy of the relative positions of the second-type frame and the first-type frame.

[0025] Preferably, the step of performing a focal loss function operation on the result of the sigmoid function in step S203 further includes: adding a sample ratio balancing factor to the focal loss function to reduce the loss of easily classified samples, and adding a sample loss adjustment factor to balance the uneven proportion of the samples themselves. Its beneficial effects are: by adding the sample loss adjustment factor to balance simple and difficult samples and reducing the loss of easily classified samples, the focal loss function pays more attention to difficult and misclassified samples in the calculation; by adding the sample ratio balancing factor to balance the uneven proportion of positive and negative samples themselves, the problem of the ordinary cross entropy loss function that the greater the output probability of the positive sample, the smaller the loss, and the smaller the output probability of the negative sample, the smaller the loss, which causes the cross entropy loss function to be slow and may not be optimized to the optimal state during the iteration process of a large number of simple samples, is solved. In addition, the computational complexity is reduced, the operation speed is improved, and while the contribution of the misclassified target individual to the loss function increases exponentially, it also takes into account the exponential decrease in the contribution of the correctly classified target individual to the loss function, so that the focal loss function reflects the overall discrimination of the feature map.

[0026] Preferably, the step of extracting comprehensive semantic information from the first-type frame and the second-type frame in step S100 to integrate the semantic information of the first-type frame and the second-type frame and outputting the first-type frame information and the second-type frame information includes:

[0027] S111: performing a loss function operation on the predicted probability values ​​of the first category boxes and the predicted probability values ​​of the second category boxes to obtain a classification joint loss result;

[0028] S112: Repeat step S111 for all candidate boxes of all grids in the neural network model that have both the first and second class boxes, and sum all the obtained classification joint loss results to obtain a total classification joint loss result. This has the beneficial effects of preventing missed detections, facilitating better extraction of comprehensive semantic information of the first and second class boxes, improving the detection rate of the first and second class boxes, effectively integrating the joint semantic information of the first and second class boxes, and improving detection accuracy.

[0029] Preferably, after executing step S112, the following steps are further performed:

[0030] S113: Performing a product operation on the total joint classification loss result and the weight coefficient of the joint classification loss to increase the joint classification loss of the first and second class boxes. This has the beneficial effect of increasing the joint classification loss of the first and second class boxes, making the calculation more accurate and improving the accuracy of the relative positions of the first and second class boxes.

[0031] Preferably, the step of classifying and identifying the first and second types of frames in the neural network model and separately defining the positions of the first and second types of frames in step S100 to obtain the positions of the first and second types of frames specifically includes:

[0032] S110: A target detection loss function is obtained by summing the classification loss function, the confidence loss function, and the regression box loss function. After constructing the target detection loss function, a backpropagation calculation is performed. This has the beneficial effect of enabling classification and recognition of the first and second class boxes in the neural network model and separately defining the positions of the first and second class boxes.

[0033] Preferably, the method for constructing the regression frame loss function in step S110 includes the following steps:

[0034] Performing a product operation on the distance and scale coefficient between the predicted center point and the true center point of the grid where the center point of the detection target is located to obtain a grid center point distance parameter, summing the grid center point distance parameters of all grids responsible for the detection target in the neural network model and all candidate boxes assigned to the detection target and then multiplying them by the coordinate box loss weight coefficient to obtain a total grid center point distance parameter;

[0035] Performing a square operation on the difference between the predicted width of the grid where the center point of the detection target is located and the actual width of the grid, and a square operation on the difference between the predicted height of the grid and the actual height of the grid, respectively, and then summing the sums, and then multiplying the sums by the proportional coefficient to obtain a grid specification difference parameter, summing the grid specification difference parameters of all grids responsible for the detection target in the neural network model and all candidate boxes assigned to the detection target, and then multiplying the sums by the coordinate box loss weight coefficient to obtain a total grid specification difference parameter;

[0036] The sum operation of the total grid center point distance parameter and the total grid specification difference parameter is performed. The beneficial effect is that the positions of the first type of frame and the second type of frame in the neural network model can be independently limited.

[0037] Preferably, the method for constructing the confidence loss function in step S110 includes the following steps:

[0038] Performing a loss function operation on the true confidence of the grid with the detected target and the predicted confidence of the grid to obtain a first confidence loss result, and summing all the first confidence loss results calculated for all candidate boxes of all grids with the detected target in the neural network model to obtain a first total confidence loss result;

[0039] Performing a loss function operation on the true confidence of a grid in the grid that is actually measured without a detection target but predicted to have a detection target and the predicted confidence of the grid to obtain a second confidence loss result, adding all the second confidence loss results calculated for all candidate boxes of all grids in the neural network model that are actually measured without a detection target but predicted to have a detection target, and then performing a product operation with the grid no detection target weight coefficient to obtain a second total confidence loss result;

[0040] The first total confidence loss result and the second total confidence loss result are summed up, which has the beneficial effect of ensuring the confidence of the grid in the neural network model, preventing missed detection, and ensuring detection accuracy.

[0041] Preferably, the method for constructing the classification loss function in step S110 includes the following steps:

[0042] Performing a loss function calculation on the true probability value of the first class box and the predicted probability value of the first class box to obtain a first class box loss result;

[0043] Performing a loss function calculation on the true probability value of the second class box and the predicted probability value of the second class box to obtain a second class box loss result;

[0044] Performing a summation operation on all first-category box loss results and all second-category box loss results calculated for all candidate boxes of all grids with detected targets in the neural network model. This has the beneficial effect of achieving accurate classification and recognition of first-category boxes and second-category boxes in the neural network model.

[0045] Preferably, before executing step S100, the following steps are further included:

[0046] S101: Annotate the image dataset and divide it into training set, validation set and test set;

[0047] S102: Perform data preprocessing on the training set, the validation set, and the test set.

[0048] Preferably, after executing step S200, the following steps are further performed:

[0049] S300: After inputting the test set into the trained neural network model to obtain a second model output result, the NMS strategy is used to optimize the second model output result to obtain a final effect, and then a compliance evaluation test is performed on the final effect to determine whether the obtained evaluation result meets the expected effect;

[0050] S400: Deploy the neural network model whose evaluation results achieve the expected effect on the chip and output the effect.

[0051] Preferably, the present invention also provides a device comprising a memory and a processor, wherein the memory stores a program, and the processor loads and executes instructions of the program. When the program is executed by the processor, the detection method is implemented.

[0052] The beneficial effect of the device of the present invention is that: a program is stored in the memory, and the processor loads and executes the instructions of the program. When the program is executed by the processor, the detection method is implemented, so that the device can be equipped to perform the detection method, and the problem of lack of relative position constraints between the second type of frame and the first type of frame is solved, thereby improving the accuracy of the relative position of the second type of frame and the first type of frame.

[0053] Preferably, the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and when the program is executed by a processor, the detection method is implemented.

[0054] The beneficial effect of the computer-readable storage medium of the present invention is that: a program is stored in the storage medium, and when the program is executed by the processor, the detection method is implemented, so that the detection method can be performed by setting a computer-readable storage medium, and the problem of lack of relative position constraints between the second type of frame and the first type of frame is solved, and the accuracy of the relative position of the second type of frame and the first type of frame is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the first relative position of the first type of frame and the second type of frame of the present invention;

[0056] Figure 2 A second schematic diagram of relative positions of the first type of frame and the second type of frame of the present invention;

[0057] Figure 3 Schematic diagram of a curve of a sigmoid function in an embodiment of the present invention;

[0058] Figure 4 Flowchart of the detection method in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with ordinary skills in the field to which the invention belongs. The words "including" and similar words used in this article mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0060] Figure 1 A schematic diagram of the first relative position of the first type of frame and the second type of frame of the present invention; Figure 2 This is a second schematic diagram of the relative positions of the first type of frame and the second type of frame of the present invention.

[0061] When performing face and figure detection, when the front of a person appears in the picture, it is hoped that both the face and the figure will be detected at the same time. The ideal effect is as follows: Figure 1 As shown, Figure 1 This is a schematic diagram of the first relative position of the face frame and the human-shaped frame of the present invention. Figure 1Indicates that the face frame 10 containing the face is within the human-shaped frame 20 containing the human figure. However, in the prior art, face and human-shaped detection are generally trained and deployed separately. Since face and human-shaped are two independent categories, their position information is not integrated during training. As a result, the relative positions of the face frame and human-shaped frame after detection are completely independent, which often results in the following error: Figure 2 The phenomenon shown, Figure 2 This is a schematic diagram showing the second relative position of the face frame and the human-shaped frame of the present invention. Figure 2 It indicates that the human-shaped frame 20 does not completely contain the face frame 10. This may lead to problems such as the inability to quickly and effectively determine whether the detected face and human shape belong to the same target. It will also be affected when the face and human shape detection results are used for other applications, resulting in low test accuracy and test errors. In addition, the existing technology lacks the integration of face and human shape semantic information during the model training process, which increases computing consumption and memory usage.

[0062] To overcome the problems existing in the prior art, an embodiment of the present invention provides a detection method, which solves the problem of lack of relative position constraints between the second type of frame and the first type of frame, and improves the accuracy of the relative position of the second type of frame and the first type of frame.

[0063] In some embodiments of the present invention, the detection method comprises the following steps:

[0064] S100: Classifying and identifying first-category boxes and second-category boxes in a neural network model and separately defining the positions of the first-category boxes and the second-category boxes to obtain first-category box positions and second-category box positions; and extracting comprehensive semantic information from the first-category boxes and the second-category boxes to integrate the semantic information of the first-category boxes and the second-category boxes and output the first-category box information and the second-category box information.

[0065] S200: constraining the relative positions of the first-type frame and the second-type frame so that the first-type frame is located within the second-type frame;

[0066] The step S200 specifically includes: determining the inclusion relationship between the first type of frame and the second type of frame, and when it is determined that the inclusion relationship between the first type of frame and the second type of frame is correct, performing data correction to reduce the weight when the relative position is correct.

[0067] In the embodiment of the present invention, the first type of frame and the second type of frame refer to the following situations: the first type of frame is part of the second type of frame, and if the first type of frame exists, the second type of frame must exist, such as plant and land detection in farms under drone scenes, human face and human shape detection, pet face and pet body detection and other application scenarios; when a part of the first type of frame is inside the second type of frame, there is no affiliation, and once coexistence, relative position constraints are required, such as pest and plant detection in pests and diseases under drone scenes, driver and car detection, cyclist and non-motor vehicle detection and other scenarios, then only the position constraints of the present invention are required.

[0068] In some embodiments of the present invention, the first type of frame is a frame that includes the pet face portion of the detection target, and the second type of frame is a frame that includes the pet face portion and the pet body portion of the detection target.

[0069] In some embodiments of the present invention, the first type of frame is a frame that includes the face of the detection target, and the second type of frame is a frame that includes both the face and body of the detection target.

[0070] In some embodiments of the present invention, the step of classifying and identifying the first and second types of boxes in the neural network model and separately limiting the positions of the first and second types of boxes to obtain the positions of the first and second types of boxes in step S100 specifically includes:

[0071] S110: The target detection loss function is obtained by performing a sum operation on the classification loss function, the confidence loss function and the regression box loss function. When there is no object in the grid in the neural network model, the target detection loss function only calculates the confidence loss function of the grid. After constructing the target detection loss function, backpropagation calculation is performed.

[0072] In some embodiments of the present invention, the method for constructing the regression box loss function in step S110 includes the following steps:

[0073] Performing a product operation on the distance and scale coefficient between the predicted center point and the true center point of the grid where the center point of the detection target is located to obtain a grid center point distance parameter, summing the grid center point distance parameters of all grids responsible for the detection target in the neural network model and all candidate boxes assigned to the detection target and then multiplying them by the coordinate box loss weight coefficient to obtain a total grid center point distance parameter;

[0074] Performing a square operation on the difference between the predicted width of the grid where the center point of the detection target is located and the actual width of the grid, and a square operation on the difference between the predicted height of the grid and the actual height of the grid, respectively, and then summing the sums, and then multiplying the sums by the proportional coefficient to obtain a grid specification difference parameter, summing the grid specification difference parameters of all grids responsible for the detection target in the neural network model and all candidate boxes assigned to the detection target, and then multiplying the sums by the coordinate box loss weight coefficient to obtain a total grid specification difference parameter;

[0075] An addition operation is performed on the total grid center point distance parameter and the total grid specification difference parameter.

[0076] Specifically, the expression of the regression box loss function is:

[0077]

[0078] Where K*K represents the grid points in the neural network model, each grid point is assigned M candidate boxes, each candidate box will get a corresponding border after passing through the network, and finally form K*K*M borders, λ coord is the weight coefficient of the coordinate frame loss, It is used to indicate whether the j-th candidate box of the i-th grid is responsible for the target. When the j-th candidate box of the i-th grid is responsible for the target, then When the jth candidate box of the i-th grid is not responsible for the target, then wi is the actual width of the i-th grid, h i is the true height of the i-th grid, x i is the x coordinate of the true center point of the i-th grid, y i is the y coordinate of the true center point of the i-th grid, is the predicted width of the i-th grid, is the predicted height of the i-th grid, is the x-coordinate of the predicted center point of the i-th grid, is the y coordinate of the predicted center point of the i-th grid. Among them, the regression box loss function is multiplied by the scale factor (2-w i *h i ) to increase the weight coefficient of the small frame and increase the loss to the small frame.

[0079] Specifically, the λ coordThe value of is generally greater than 1, and the true width of the i-th grid, the true height of the i-th grid, the x-coordinate of the true center point of the i-th grid, the y-coordinate of the true center point of the i-th grid, the predicted width of the i-th grid, the predicted height of the i-th grid, the x-coordinate of the predicted center point of the i-th grid, and the y-coordinate of the predicted center point of the i-th grid are all 0-1.

[0080] In some possible embodiments of the present invention, the weight coefficient of the coordinate frame loss is set to a value of 1.1-1.5 to increase the coordinate frame loss.

[0081] In some embodiments of the present invention, the method for constructing the confidence loss function in step S110 includes the following steps:

[0082] Performing a loss function operation on the true confidence of the grid with the detected target and the predicted confidence of the grid to obtain a first confidence loss result, and summing all the first confidence loss results calculated for all candidate boxes of all grids with the detected target in the neural network model to obtain a first total confidence loss result;

[0083] Performing a loss function operation on the true confidence of a grid in the grid that is actually measured without a detection target but predicted to have a detection target and the predicted confidence of the grid to obtain a second confidence loss result, adding all the second confidence loss results calculated for all candidate boxes of all grids in the neural network model that are actually measured without a detection target but predicted to have a detection target, and then performing a product operation with the grid no detection target weight coefficient to obtain a second total confidence loss result;

[0084] Perform a sum operation on the first total confidence loss result and the second total confidence loss result.

[0085] Specifically, the expression of the confidence loss function is:

[0086]

[0087] in, It is used to indicate whether the j-th candidate box of the i-th grid is not responsible for the target. When the j-th candidate box of the i-th grid is not responsible for the target, then When the jth candidate box of the i-th grid is responsible for the target, then λ noobj is the weight coefficient of the grid without detection target, C i is the true confidence of the candidate box, is the prediction confidence of the candidate box. When the bounding box of the i-th grid point is responsible for predicting a target, the prediction confidence of the candidate box is 1. When the bounding box of the i-th grid point is not responsible for predicting a target, the prediction confidence of the candidate box is 0. Specifically, the confidence loss function is divided into two parts, namely the target confidence loss function and the non-target confidence loss function. When there is no object in the grid in the neural network model, the target detection loss function only calculates the confidence loss function of the grid. Among them, the non-target confidence loss function also adds a grid no-detection target weight coefficient to reduce the contribution weight of the non-target calculation part.

[0088] In some possible embodiments of the present invention, the grid-no-detection-target weight coefficient is set to a value of 0.5-1.1 to reduce the contribution weight to the calculation of grids in which no detection targets are actually measured but detection targets are predicted. In some specific embodiments of the present invention, the grid-no-detection-target weight coefficient is set to a value of 0.5.

[0089] In some embodiments of the present invention, the method for constructing the classification loss function in step S110 includes the following steps:

[0090] Performing a loss function calculation on the true probability value of the first class box and the predicted probability value of the first class box to obtain a first class box loss result;

[0091] Performing a loss function calculation on the true probability value of the second class box and the predicted probability value of the second class box to obtain a second class box loss result;

[0092] Perform a sum operation on all first-category box loss results and all second-category box loss results calculated for all candidate boxes of all grids with detection targets in the neural network model.

[0093] Specifically, the expression of the classification loss function is:

[0094]

[0095] Among them, p i is the true probability value of the i-th grid in the feature map output by the network model, is the predicted probability value of the i-th grid in the feature map output by the network model, and c represents the category, which includes the second-category box and the first-category box. Specifically, when the j-th candidate box in the i-th grid is responsible for predicting a real object, the bounding box generated by this candidate box will be used to calculate the classification loss.

[0096] In some specific embodiments of the present invention, the expression of the target detection loss function is:

[0097]

[0098] In some embodiments of the present invention, step S200 further includes determining an inclusion relationship between the first-category box and the second-category box, and when it is determined that the inclusion relationship between the first-category box and the second-category box is correct, performing data correction to reduce the weight when the relative position is correct includes:

[0099] S201: performing normalization operations on the center point, width, and height of the predicted first-category box and the predicted second-category box, respectively, to obtain the upper-left point coordinates and the lower-right point coordinates of the approximate first-category box and the approximate second-category box, respectively;

[0100] S202: performing horizontal and vertical comparison operations on the obtained upper left point coordinates and lower right point coordinates of the approximate first category frame and the upper left point coordinates and lower right point coordinates of the approximate second category frame respectively to obtain comparison parameters, and performing a sigmoid function operation on the comparison parameters to obtain a sigmoid function result;

[0101] S203: Perform a focal loss function operation on the sigmoid function result and then perform a sum operation to obtain a regression loss result of the relative position of the first type frame and the second type frame.

[0102] In the embodiment of the present invention, the border is formed by connecting and enclosing line segments passing through the upper left point, the upper right point, the lower right point and the lower left point in sequence, the coordinates of the upper left point and the coordinates of the lower right point are the coordinates of the diagonal points in the border, and the border includes a first type of frame and a second type of frame.

[0103] In some specific embodiments of the present invention, the step of performing normalization operations on the center point, width, and height of the predicted first-category box and the predicted second-category box in step S201 to obtain the coordinates of the upper left point and the coordinates of the lower right point of the approximate first-category box and the approximate second-category box, respectively, includes performing the following steps:

[0104] Set the center points of the predicted first-class box and the predicted second-class box with index i to be Width Gao Wei The c represents any one of face and human, the face is a first-category frame, and the human is a second-category frame;

[0105] The predicted first type frame and the predicted second type frame are Respectively corresponding to any one of the first type frame and the second type frame Perform normalization processing, and convert the first type frame and the second type frame into The first and second types of frames are respectively Normalization is performed to obtain the coordinates of the upper left point (xi1c, y i1c ) and the coordinates of the lower right point (x i2c ,y i2c ).

[0106] Specifically, the coordinates (x i1c ,y i1c ) and the lower right point coordinates (x i2c ,y i2c ) are as follows:

[0107]

[0108]

[0109]

[0110]

[0111] Among them, the upper left point is set as the reference point, and the specific setting method is conventional technology in this field and will not be repeated here.

[0112] Figure 3 Schematic diagram of a curve of the sigmoid function in an embodiment of the present invention.

[0113] In some embodiments of the present invention, the expression of the sigmoid function is:

[0114]

[0115] Among them, P t is the result obtained after performing sigmoid function calculation, and Z is the comparison parameter.

[0116] In some embodiments of the present invention, the contrast parameters in step S202 include a first horizontal contrast parameter, a second horizontal contrast parameter, a first vertical contrast parameter, and a second vertical contrast parameter;

[0117] The first horizontal contrast parameter is obtained by subtracting the x-coordinate value of the upper left point of the approximate second type frame from the x-coordinate value of the upper left point of the approximate first type frame;

[0118] The first vertical contrast parameter is obtained by subtracting the y coordinate value of the upper left point of the approximate second type frame from the y coordinate value of the upper left point of the approximate first type frame;

[0119] The second horizontal contrast parameter is obtained by subtracting the x-coordinate value of the lower right point of the approximate second type frame from the x-coordinate value of the lower right point of the approximate second type frame;

[0120] The second vertical contrast parameter is obtained by subtracting the y coordinate value of the lower right point of the approximate second type frame from the y coordinate value of the lower right point of the approximate second type frame.

[0121] Specifically, the expressions of the first horizontal contrast parameter Z1, the first vertical contrast parameter Z2, the second horizontal contrast parameter Z3 and the second vertical contrast parameter Z4 are respectively:

[0122] Z1=x ilA -x i1B

[0123] Z2=y ilA -y i1B

[0124] Z3=x i2B -x i2A

[0125] Z4=y i2B -y i2A

[0126] Among them, x i1A is the x coordinate value of the upper left point of the approximate first-class box with index i, x i1B is the x coordinate value of the upper left point of the approximate second type box with index i, y i1A is the y coordinate value of the upper left point of the approximate first-class box with index i, y i1B is the y coordinate value of the upper left point of the approximate second type box with index i, x i2A is the x-coordinate value of the lower right point of the approximate first-class box with index i, x i2B is the x-coordinate value of the lower right point of the approximate second-class box with index i, i2A is the y coordinate value of the lower right point of the approximate first-class box with index i, y i2B is the y-coordinate value of the lower right point of the approximate second-category box with index i.

[0127] refer to Figure 3 , the calculated P t The range is 0-1, when the P t The value of satisfies [0.5, 1], indicating that the first type of frame is inside the second type of frame, that is, the second type of frame completely contains the first type of frame. When the P tThe value of satisfies (0, 0.5), indicating that the first category box is not completely inside the second category box. This feature is used to apply the focal loss function to the result of the sigmoid function to reduce the weight when the relative position is correct, so that the model can focus more on correcting the boxes with incorrect relative positions during training.

[0128] In some embodiments of the present invention, the step of performing a focal loss function operation on the sigmoid function result in step S203 also includes: adding a sample ratio balance factor in the focal loss function to reduce the loss of easy-to-classify samples, and adding a sample loss adjustment factor to balance the uneven proportion of the samples themselves.

[0129] Specifically, the expression of the focal loss function is:

[0130] fl(P t )=-α t (1-P t ) γ log(P t )

[0131] Among them, fl(P t ) is the focal loss function, α t is the sample ratio balance factor, and γ is the sample loss adjustment factor.

[0132] In some possible embodiments of the present invention, the value of the sample loss adjustment factor is greater than 0, and the value of the sample ratio balance factor is 0.1-0.9. The sample loss adjustment factor being greater than 0 can effectively reduce the loss of easily classified samples, so that the focal loss function pays more attention to difficult and misclassified samples in the calculation. The value of the sample ratio balance factor is 0.1-0.9, which balances the uneven proportion of positive and negative samples. A sample ratio balance factor value greater than 0.9 will result in an excessive proportion of positive samples, and a sample ratio balance factor value less than 0.1 will result in an excessive proportion of negative samples. In some specific embodiments of the present invention, the sample loss adjustment factor is 2, and the sample ratio balance factor is 0.25.

[0133] In some embodiments of the present invention, the regression loss result of the relative position of the first type box and the second type box in step S200 is obtained by calculating the regression loss function of the relative position of the first type box and the second type box. The expression of the regression loss function of the relative position of the first type box and the second type box is:

[0134] loss(dist i )=fl(sigmoid(x i1A -x i1B))+fl(sigmoid(y i1A -y i1B ))+

[0135] fl(sigmoid(x i2B -x i2A ))+fl(sigmoid(y i2B -y i2A ))

[0136] Among them, x i1A -x i1B 、y i1A -y i1B 、x i2B -x i2A 、y i2B -y i2A The comparison parameters obtained by performing horizontal and vertical comparisons on the upper left point coordinates and lower right point coordinates of the approximate first type frame with the upper left point coordinates and lower right point coordinates of the approximate second type frame, x i1A is the x coordinate value of the upper left point of the approximate first-class box with index i, x i1B is the x coordinate value of the upper left point of the approximate second type box with index i, y i1A is the y coordinate value of the upper left point of the approximate first-class box with index i, y i1B is the y coordinate value of the upper left point of the approximate second type box with index i, x i2A is the x-coordinate value of the lower right point of the approximate first-class box with index i, x i2B is the x-coordinate value of the lower right point of the approximate second-class box with index i, i2A is the y coordinate value of the lower right point of the approximate first-class box with index i, y i2B is the y-coordinate value of the lower right point of the approximate second-category box with index i.

[0137] In some embodiments of the present invention, after step S203 is executed, the following steps are further included:

[0138] S204: Repeat steps S201 to S203 for all candidate frames of all grids that have both the first and second class frames in the neural network model, that is, perform the regression loss function operation of the relative position of the first and second class frames, and sum up all the obtained regression loss results of the relative positions of the first and second class frames to obtain the total regression loss result of the relative positions of the first and second class frames, so as to prevent missed detection and improve the accuracy of the relative position of the second class frames and the first class frames.

[0139] In some embodiments of the present invention, after executing step S204, the following steps may be further performed:

[0140] S205: Perform a multiplication operation on the total regression loss result of the relative position of the first-category frame and the second-category frame and the weight coefficient of the relative position loss of the first-category frame and the second-category frame to increase the relative position loss of the first-category frame and the second-category frame.

[0141] In some possible embodiments of the present invention, the weight coefficient of the relative position loss of the first type of frame and the second type of frame is in the range of 1.1-1.5, so as to increase the loss of the relative position of the first type of frame and the second type of frame, make the calculation more accurate, and improve the accuracy of the relative position of the first type of frame and the second type of frame.

[0142] In some embodiments of the present invention, the step of extracting comprehensive semantic information from the first-type frame and the second-type frame in step S100 to integrate the semantic information of the first-type frame and the second-type frame and outputting the first-type frame information and the second-type frame information includes:

[0143] S111: performing a loss function operation on the predicted probability values ​​of the first category boxes and the predicted probability values ​​of the second category boxes to obtain a classification joint loss result;

[0144] S112: Repeat step S111 for all candidate frames of all grids that have both the first class frame and the second class frame in the neural network model, that is, perform the classification joint loss function operation of the first class frame and the second class frame, and add up all the obtained classification joint loss results to obtain a total classification joint loss result, so as to prevent missed detection, facilitate better extraction of the comprehensive semantic information of the first class and the second class, improve the detection rate of the first class frame and the second class frame, effectively integrate the joint semantic information of the first class and the second class, and improve the detection accuracy.

[0145] In some embodiments of the present invention, after executing step S112, the following steps may be further performed:

[0146] S113: Perform a multiplication operation on the total classification joint loss result and the weight coefficient of the classification joint loss to increase the classification joint loss of the first category box and the second category box.

[0147] In some possible embodiments of the present invention, the weight coefficient of the classification joint loss ranges from 1.0 to 1.2 to increase the classification joint loss of the first category frame and the second category frame, making the calculation more accurate and improving the accuracy of the relative position of the first category and the second category.

[0148] Specifically, the expression of the joint loss function of the first and second category box classification is:

[0149]

[0150] Wherein, K*K represents the grid points in the neural network model, is the predicted probability value of the first class box in the i-th grid in the feature map output by the network model, is the predicted probability value of the second class box in the i-th grid in the feature map output by the network model, Indicates that the neural network model contains both the first type of frame and the second type of frame. Considering that the second type of frame must exist under the premise of the first type of frame, when When it is 1, the joint loss function of the first and second category box classification is increased accordingly, which can better extract the comprehensive semantic information of the two categories.

[0151] In some specific embodiments of the present invention, the expressions of the detection loss functions of the first and second type boxes are:

[0152]

[0153] Among them, λ co-coord is the weight coefficient of the relative position loss between the first and second type boxes, λ co-class The weight coefficient for the joint classification loss of the first and second class boxes.

[0154] In some embodiments of the present invention, the following steps are further included before step S100 is performed:

[0155] S101: Annotate the image dataset and divide it into training set, validation set and test set;

[0156] S102: Perform data preprocessing on the training set, the validation set, and the test set.

[0157] In some embodiments of the present invention, after executing step S200, the following steps are further performed:

[0158] S300: After inputting the test set into the trained neural network model to obtain a second model output result, the NMS strategy is used to optimize the second model output result to obtain a final effect, and then a compliance evaluation test is performed on the final effect to determine whether the obtained evaluation result meets the expected effect;

[0159] S400: Deploy the neural network model whose evaluation results achieve the expected effect on the chip and output the effect.

[0160] Figure 4 Flowchart of the detection method in an embodiment of the present invention.

[0161] In some specific embodiments of the present invention, reference Figure 4 The detection method includes a face and human shape detection method, and the face and human shape detection method includes the following steps:

[0162] S10: Label the human image dataset and face image dataset and divide them into training set, validation set and test set;

[0163] S20: performing data preprocessing on the training set, the validation set, and the test set.

[0164] S30: constructing a face and shape detection loss function, using the face and shape detection loss function to perform model training on a neural network model to obtain a trained neural network model, and using the trained neural network model to perform face and shape detection;

[0165] S40: After inputting the test set into the trained neural network model to obtain a second model output result, the NMS strategy is used to optimize the second model output result to obtain a final effect, and then a compliance evaluation test is performed on the final effect to determine whether the obtained evaluation result achieves the expected effect;

[0166] S50: Deploy the neural network model whose evaluation results achieve the expected effect on the chip and output the effect.

[0167] Specifically, the face and shape detection method is also applicable to the detection of drivers and vehicles in a driving state. The specific steps are not repeated here.

[0168] In some embodiments of the present invention, the step S10 of annotating the image dataset and dividing it into a training set, a validation set, and a test set includes: collecting an image dataset formed by images of different environments, different backgrounds, different postures, and different positions taken by a camera in a real environment, and using an annotation tool to generate the annotation box position and label information of the corresponding target, where the label information of the human category is human and face; dividing the annotated image dataset into a training set, a validation set, and a test set respectively; generating a list of the training set images and shuffling the arrangement order; clustering the target boxes corresponding to the labels in all images to generate 12 cluster points.

[0169] In some embodiments of the present invention, the data preprocessing of the training set, the validation set, and the test set in step S20 includes performing a normalization operation using the RGB channels of the image, and performing an operation as shown in the following formula on each channel:

[0170] X i R, G, B

[0171] R, G, and B represent the red, green, and blue channels, respectively. The RGB color model is an industry standard for color. It creates a wide variety of colors by varying and superimposing the three color channels (red (R), green (G), and blue (B). RGB represents the colors of the three channels. This standard encompasses nearly all colors perceptible by human vision and is currently one of the most widely used color systems. Normalization using the RGB channels of an image is a common technique in the field and will not be detailed here.

[0172] In some embodiments of the present invention, after the normalization operation is performed in step S20, the image is randomly flipped horizontally, the image is cropped to the minimum area containing the target image, the saturation multiple is randomly adjusted within [1 / 1.5, 1.5], the exposure multiple is randomly adjusted within [1 / 1.5, 1.5], the hue multiple is randomly adjusted within [1 / 1.2, 1.2], and the image is randomly rotated at an angle of [-30, 30] according to the center point; the random probability of all the above is 50%.

[0173] In some embodiments of the present invention, the step S30 specifically includes the following steps: after the neural network model is trained on the training set for several generations using the face and human shape detection loss function, the validation set is input into the neural network model to obtain a first model output result, and then the first model output result is optimized using the NMS strategy, and then a trained neural network model is obtained based on the optimized first model output result. In some preferred embodiments of the present invention, the first model output result is optimized using the NMS strategy to evaluate the performance of the neural network model, and when the performance of the neural network model is poor, the model is optimized to finally obtain a trained neural network model. The model optimization includes adjusting the neural network structure for the human shape network training model, and the adjustment refers to expanding the number of convolution kernels by a multiple of 1.25, and expanding the training set by adding image data of the scene, and then performing model training to achieve the purpose of optimizing the model, and finally obtaining a trained neural network model. The model optimization is common knowledge in the field and will not be described in detail here.

[0174] In some embodiments of the present invention, the compliance assessment test in step S40 is to use a camera to conduct scene measurements on the test set used for training, and compare them with the selected products on the market. Figure 4 , the effect is better than the average value of the effects of the selected products on the market. If it is judged that the evaluation result has achieved the expected effect, step S50 is performed to deploy the neural network model whose evaluation result has achieved the expected effect on the chip to output the effect. Otherwise, the processing of steps S100-S400 is repeated.

[0175] In some embodiments of the present invention, the neural network model is any one of YOLOv3, YOLOv4 and YOLOv5s. YOLO (You Only Look Once) is an object recognition and positioning algorithm based on a deep neural network. Its biggest feature is that it runs very fast and can be used in real-time systems.

[0176] In some embodiments of the present invention, a device is further provided, including a memory and a processor, wherein the memory stores a program, and the processor loads and executes instructions of the program. When the program is executed by the processor, the detection method is implemented.

[0177] In some embodiments of the present invention, a computer-readable storage medium is further provided, wherein the storage medium stores a program, and when the program is executed by a processor, the detection method is implemented.

[0178] While the embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations of these embodiments are possible. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as set forth in the claims. Furthermore, the invention described herein is susceptible to other embodiments and may be practiced or implemented in a variety of ways.

Claims

1. A detection method, characterized in that: The following steps are involved: S100: Classifying and identifying first-category boxes and second-category boxes in a neural network model and separately defining the positions of the first-category boxes and the second-category boxes to obtain first-category box positions and second-category box positions; and extracting comprehensive semantic information from the first-category boxes and the second-category boxes to integrate the semantic information of the first-category boxes and the second-category boxes and output the first-category box information and the second-category box information. S200: constraining the relative positions of the first-type frame and the second-type frame so that the first-type frame is located within the second-type frame; The step S200 specifically includes: determining the inclusion relationship between the first-category frame and the second-category frame, and when it is determined that the inclusion relationship between the first-category frame and the second-category frame is correct, performing data correction to reduce the weight when the relative position is correct, including: S201: performing normalization operations on the center point, width, and height of the predicted first-category box and the predicted second-category box, respectively, to obtain the upper-left point coordinates and the lower-right point coordinates of the approximate first-category box and the approximate second-category box, respectively; S202: performing horizontal and vertical comparison operations on the obtained upper left point coordinates and lower right point coordinates of the approximate first category frame and the upper left point coordinates and lower right point coordinates of the approximate second category frame respectively to obtain comparison parameters, and performing a sigmoid function operation on the comparison parameters to obtain a sigmoid function result; S203: performing a focal loss function operation on the sigmoid function result and then performing a sum operation to obtain a regression loss result of the relative position of the first type frame and the second type frame; S204: Repeat steps S201 to S203 for all candidate boxes of all grids that have both the first and second class boxes in the neural network model, and sum the obtained regression loss results of the relative positions of all the first and second class boxes to obtain a total regression loss result of the relative positions of the first and second class boxes; S205: Perform a multiplication operation on the total regression loss result of the relative position of the first-category frame and the second-category frame and the weight coefficient of the relative position loss of the first-category frame and the second-category frame to increase the relative position loss of the first-category frame and the second-category frame.

2. The detection method according to claim 1, wherein The contrast parameters in step S202 include a first horizontal contrast parameter, a second horizontal contrast parameter, a first vertical contrast parameter, and a second vertical contrast parameter; The first horizontal contrast parameter is obtained by subtracting the x-coordinate value of the upper left point of the approximate second type frame from the x-coordinate value of the upper left point of the approximate first type frame; The first vertical contrast parameter is obtained by subtracting the y coordinate value of the upper left point of the approximate second type frame from the y coordinate value of the upper left point of the approximate first type frame; The second horizontal contrast parameter is obtained by subtracting the x-coordinate value of the lower right point of the approximate second type frame from the x-coordinate value of the lower right point of the approximate second type frame; The second vertical contrast parameter is obtained by subtracting the y coordinate value of the lower right point of the approximate second type frame from the y coordinate value of the lower right point of the approximate second type frame.

3. The detection method according to claim 1, wherein The step of performing a focal loss function operation on the sigmoid function result in step S203 further includes: adding a sample ratio balancing factor in the focal loss function to reduce the loss of easily classified samples, and adding a sample loss adjustment factor to balance the uneven proportion of the samples themselves.

4. The detection method according to claim 1, wherein The steps of extracting comprehensive semantic information from the first-type frame and the second-type frame in step S100, integrating the semantic information of the first-type frame and the second-type frame, and outputting the first-type frame information and the second-type frame information include: S111: performing a loss function operation on the predicted probability values ​​of the first category boxes and the predicted probability values ​​of the second category boxes to obtain a classification joint loss result; S112: Repeat step S111 for all candidate boxes of all grids in the neural network model that have both the first category box and the second category box, and add up all the obtained classification joint loss results to obtain a total classification joint loss result.

5. The detection method according to claim 4, characterized in that After step S112 is executed, the following steps are further included: S113: Perform a multiplication operation on the total classification joint loss result and the weight coefficient of the classification joint loss to increase the classification joint loss of the first category box and the second category box.

6. The detection method according to claim 1, characterized in that The steps of classifying and identifying the first and second types of frames in the neural network model and separately defining the positions of the first and second types of frames in step S100 to obtain the positions of the first and second types of frames specifically include: S110: Obtain a target detection loss function by performing a sum operation on the classification loss function, the confidence loss function, and the regression box loss function. After constructing the target detection loss function, perform a backpropagation calculation.

7. The detection method according to claim 6, characterized in that The method for constructing the regression frame loss function in step S110 includes the following steps: Performing a product operation on the distance and scale coefficient between the predicted center point and the true center point of the grid where the center point of the detection target is located to obtain a grid center point distance parameter, summing the grid center point distance parameters of all grids responsible for the detection target in the neural network model and all candidate boxes assigned to the detection target and then multiplying them by the coordinate box loss weight coefficient to obtain a total grid center point distance parameter; Performing a square operation on the difference between the predicted width of the grid where the center point of the detection target is located and the actual width of the grid, and a square operation on the difference between the predicted height of the grid and the actual height of the grid, respectively, and then summing the sums, and then multiplying the sums by the proportional coefficient to obtain a grid specification difference parameter, summing the grid specification difference parameters of all grids responsible for the detection target in the neural network model and all candidate boxes assigned to the detection target, and then multiplying the sums by the coordinate box loss weight coefficient to obtain a total grid specification difference parameter; An addition operation is performed on the total grid center point distance parameter and the total grid specification difference parameter.

8. The detection method according to claim 6, characterized in that The method for constructing the confidence loss function in step S110 includes the following steps: Performing a loss function operation on the true confidence of the grid with the detected target and the predicted confidence of the grid to obtain a first confidence loss result, and summing all the first confidence loss results calculated for all candidate boxes of all grids with the detected target in the neural network model to obtain a first total confidence loss result; Performing a loss function operation on the true confidence of a grid in the grid that is actually measured without a detection target but predicted to have a detection target and the predicted confidence of the grid to obtain a second confidence loss result, adding all the second confidence loss results calculated for all candidate boxes of all grids in the neural network model that are actually measured without a detection target but predicted to have a detection target, and then performing a product operation with the grid no detection target weight coefficient to obtain a second total confidence loss result; Perform a sum operation on the first total confidence loss result and the second total confidence loss result.

9. The detection method according to claim 6, characterized in that The method for constructing the classification loss function in step S110 includes the following steps: Performing a loss function calculation on the true probability value of the first class box and the predicted probability value of the first class box to obtain a first class box loss result; Performing a loss function calculation on the true probability value of the second class box and the predicted probability value of the second class box to obtain a second class box loss result; Perform a sum operation on all first-category box loss results and all second-category box loss results calculated for all candidate boxes of all grids with detection targets in the neural network model.

10. The detection method according to claim 1, characterized in that Before step S100 is executed, the following steps are also included: S101: Annotate the image dataset and divide it into training set, validation set and test set; S102: Perform data preprocessing on the training set, the validation set, and the test set.

11. The detection method according to claim 10, characterized in that: After step S200 is executed, the following steps are also included: S300: After inputting the test set into the trained neural network model to obtain a second model output result, the NMS strategy is used to optimize the second model output result to obtain a final effect, and then a compliance evaluation test is performed on the final effect to determine whether the obtained evaluation result meets the expected effect; S400: Deploy the neural network model whose evaluation results achieve the expected effect on the chip and output the effect.

12. A device, characterized in that The invention comprises a memory and a processor, wherein a program is stored in the memory, and the processor loads and executes instructions of the program. When the program is executed by the processor, the detection method according to any one of claims 1 to 11 is implemented.

13. A computer-readable storage medium, characterized in that The storage medium stores a program, and when the program is executed by the processor, the detection method according to any one of claims 1 to 11 is implemented.

Images