Target detection method and device, electronic equipment and computer readable storage medium
By weighted average processing of the object detection box filtered by the NMS model, the problem of low accuracy of the detection box is solved, and higher object detection accuracy is achieved.
Patent Information
- Application Number
- CN202311047655.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-18
- Publication Date
- 2025-02-21
AI Technical Summary
After the existing object detection method is filtered by the prediction box, the accuracy of the detection box is not high, thereby reducing the accuracy of the object detection.
By weighted averaged the prediction boxes of the same target object filtered by the NMS model, a more accurate set of detection boxes is generated for object detection.
Improve the accuracy and recall of the detection frame, thereby improving the accuracy of target detection.
Smart Images
Figure CN119494951A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to a target detection method, device, electronic device, and computer-readable storage medium. Background Art
[0002] In target detection, accurate determination of the target detection box is particularly important for target detection.
[0003] In the related art, when performing target detection, multiple prediction boxes of the target object are usually predicted through a prediction model based on a neural network algorithm, and then the prediction boxes are filtered through the non-maximum suppression (NMS) algorithm in post-processing, and the remaining prediction boxes are directly used as the detection results.
[0004] Usually after filtering by the NMS algorithm, there will be multiple detection boxes corresponding to each target object. The accuracy of each detection box may not be high, which will lead to inaccurate target detection. Summary of the invention
[0005] In view of this, the present application provides a target detection method, device, electronic device and computer-readable storage medium; the obtained detection frame has better precision and recall, thereby improving the accuracy of target detection.
[0006] The technical solution of the present disclosure is achieved as follows:
[0007] In one embodiment, a target detection method is provided, the method comprising:
[0008] Get image data;
[0009] Inputting the image data into a prediction model, obtaining a prediction frame of a target object associated with the image data through the prediction model to form a first prediction frame set;
[0010] Filter the first prediction box set based on the NMS model to obtain a second prediction box set of the target object;
[0011] Performing weighted averaging processing on the prediction frames of the same target object in the second prediction frame set to obtain a detection frame set of the image data;
[0012] Perform object detection on the image data based on the detection frame set.
[0013] Further, performing weighted averaging processing on the prediction frames of the same target object in the second prediction frame set to obtain the detection frame set of the image data includes:
[0014] Select the prediction box with the highest confidence in the current second prediction box set as the candidate box;
[0015] Calculating the intersection-over-union ratios of the candidate box and other prediction boxes in the second prediction box set respectively;
[0016] If there is an IoU greater than a first preset threshold value among the calculated IoUs, the prediction box corresponding to the IoU greater than the first preset threshold value and the candidate box are weighted averaged to obtain a detection box of the target object, and the detection box is added to the detection box set; wherein the first preset threshold value is less than the second preset threshold value, and the second preset threshold value is the threshold value used by the NMS model;
[0017] Furthermore, the method further comprises:
[0018] If there is no intersection-over-union ratio greater than the first preset threshold value among the calculated intersection-over-union ratios, the candidate frame is used as the detection frame of the target object and added to the detection frame set.
[0019] Furthermore, the method further comprises:
[0020] If the prediction frame corresponding to the intersection-and-union ratio greater than the first preset threshold value and the candidate frame are subjected to weighted averaging processing, the prediction frame corresponding to the intersection-and-union ratio greater than the first preset threshold value and the candidate frame are deleted from the second prediction frame set;
[0021] or,
[0022] If there is no IoU ratio greater than the first preset threshold value among the calculated IoU ratios, the candidate frame is deleted from the second prediction frame set.
[0023] Furthermore, the method further comprises:
[0024] Determine whether the second prediction frame set is currently empty;
[0025] If it is not empty, the step of selecting the prediction frame with the highest confidence in the current second prediction frame set as the candidate frame and subsequent steps is performed until the second prediction frame set is empty.
[0026] Furthermore, when the weighted average processing is performed, the weight corresponding to the i-th prediction box is: the product of the confidence of the i-th prediction box and the corresponding intersection-and-union ratio; wherein the intersection-and-union ratio corresponding to the i-th prediction box is the intersection-and-union ratio of the i-th prediction box and the candidate box.
[0027] further,
[0028] If there are target objects of different categories, the second prediction box sets are obtained for target objects of different categories respectively.
[0029] In another embodiment, a target detection device comprises:
[0030] An acquisition unit, configured to acquire image data;
[0031] A prediction unit is configured to input the image data into a prediction model, and obtain a prediction box of a target object associated with the image data through the prediction model to form a first prediction box set;
[0032] A filtering unit, configured to filter the first prediction box set based on the NMS model to obtain a second prediction box set of the target object;
[0033] a processing unit configured to perform weighted averaging processing on the prediction frames of the same target object in the second prediction frame set to obtain a detection frame set of the image data;
[0034] A detection unit is configured to perform target detection on the image data based on the detection frame set.
[0035] In another embodiment, an electronic device is provided, comprising:
[0036] processor;
[0037] a memory for storing executable instructions for the processor;
[0038] The processor is configured to execute the executable instructions to implement the target detection method as described in any one of the above items.
[0039] In another embodiment, a computer-readable storage medium, when at least one instruction in the computer-readable storage medium is executed by a processor of an electronic device, enables the electronic device to implement any of the target detection methods described above.
[0040] It can be seen from the above technical solutions that the target detection method, device, electronic device and computer-readable storage medium of the embodiments of the present disclosure, after using the NMS model to filter the prediction frame, do not directly use the filtered prediction frame for target detection, but perform weighted averaging processing on the filtered prediction frames of the same target object to obtain the detection frame of the corresponding target object, and then perform target detection; this makes the obtained detection frame have better precision and recall, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0042] Figure 1 is a flow chart of a target detection method according to an exemplary embodiment;
[0043] Figure 2 is a flowchart showing a method of obtaining a detection frame set of a target object based on a second prediction frame set according to a specific embodiment;
[0044] Figure 3 is another flowchart of obtaining a detection frame set of a target object based on a second prediction frame set according to a specific embodiment;
[0045] Figure 4 is a flow chart of another target detection method according to an exemplary embodiment;
[0046] Figure 5 is a schematic diagram showing a comparison of a prediction box and a detection box corresponding to a target object according to an exemplary embodiment;
[0047] Figure 6 is a schematic diagram of a target detection device according to an exemplary embodiment;
[0048] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0049] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and examples.
[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0051] In object detection, in order to improve the accuracy of object detection, it is very important to improve the accuracy of determining the detection box of the target object.
[0052] In the related art, when performing target detection, multiple prediction boxes of the target object are usually predicted through a prediction model based on a neural network algorithm, and then the prediction boxes are filtered through the NMS algorithm in post-processing, and the remaining prediction boxes are directly used as detection results; among them, when using the NMS algorithm to filter redundant prediction boxes, the prediction boxes are first sorted according to confidence, and the prediction box with the largest confidence is selected as the candidate box, and the intersection and union ratio is calculated with other prediction boxes, and the prediction boxes with an intersection and union ratio greater than a threshold are filtered out.
[0053] The solution for determining the detection frame in the related art may lead to the following two situations:
[0054] First, when there are multiple prediction boxes for a target true box, the one with a more accurate prediction box, that is, the one with a larger intersection-over-intersection ratio with the true (GT, Ground Truth) box, may also have a lower corresponding confidence. In this case, when the more accurate prediction box is filtered by the NMS algorithm, it is likely that it will not be selected as the optimal box due to confidence reasons and will be suppressed by the NMS algorithm. The main reason is that NMS initially sorts according to confidence.
[0055] The second type is that after filtering by the NMS algorithm, there are still multiple prediction boxes left for a target object, and all of the prediction boxes may not be accurate.
[0056] For example, after filtering by the NMS algorithm, two prediction boxes are output for a target object; one prediction box has a high IoU with the GT box and is kept as a positive detection; the other prediction box has a low IoU with the GT box and is usually regarded as a false detection. But in reality, both prediction boxes are not very accurate.
[0057] The detection frame obtained by the technical solution of directly outputting the detection frame after filtering by the NMS algorithm is not accurate, resulting in low target detection accuracy. Based on this, the embodiments of the present disclosure provide a target detection method, device, electronic device and computer-readable storage medium, so that the accuracy and recall of the obtained detection frame are better, thereby improving the accuracy of target detection.
[0058] Figure 1 is a flow chart of a target detection method according to an exemplary embodiment. Figure 1 As shown, the target detection method of the embodiment of the present disclosure mainly includes the following steps 101 to 104.
[0059] Step 101, acquiring image data.
[0060] The image data may be acquired in real time, or may be data acquired in advance and stored, may be a single image data, or may be multiple continuous or discontinuous image frames; this is not limited in the embodiments of the present application. The stored data may be data stored in any form in any storage device.
[0061] Step 102: input the image data into a prediction model, and obtain a prediction box of a target object associated with the image data through the prediction model to form a first prediction box set.
[0062] Obtaining the first prediction box set in this step can be achieved through a prediction model. The prediction model can be a model established based on a neural network algorithm. However, in the specific implementation, there is no restriction on the method of obtaining the first prediction box set.
[0063] For each target object in the image data, multiple prediction boxes and the confidence corresponding to each prediction box are predicted. The confidence and the prediction box can be combined into a first prediction box set, or only the prediction box can be combined into the first prediction box set, and the execution degree of each prediction box can be stored through other sets or any other way that can be represented and found.
[0064] Step 103: Filter the first prediction box set based on the NMS model to obtain a second prediction box set of the target object.
[0065] The NMS model is established based on the NMS algorithm. The process of filtering the prediction box by the NMS model can be:
[0066] The first step is to sort each prediction box in the first prediction set according to the confidence level, and select the prediction box with the highest confidence level as the candidate box;
[0067] The second step is to traverse the remaining prediction boxes. If a prediction box whose intersection-over-union ratio with the candidate box is greater than a second preset threshold value is found, the corresponding prediction box is deleted.
[0068] The third step is to select the prediction box with the highest confidence from the unprocessed prediction box as the candidate box again, and repeat the second step until all prediction boxes are processed.
[0069] The second prediction box set of the target object obtained after filtering by the NMS model will usually include multiple prediction boxes, but the number of prediction boxes corresponding to each target object will usually not exceed 5, as well as the confidence of each prediction box; this is just a conventional explanation. In the embodiments of the present application, there is no limit on the number of prediction boxes corresponding to each target object after filtering.
[0070] In the embodiment of the present application, there is no restriction on the filtering process of the prediction box based on the NMS model, and it can be implemented according to the relevant procedures.
[0071] When implementing the NMS model, it is very important to set the threshold value, which is not limited in the embodiment of the present application, and can be set to 0.65, for example.
[0072] If there are target objects of different categories, second prediction box sets are obtained for target objects of different categories respectively; and different second prediction boxes are processed respectively in subsequent processing.
[0073] If the target objects include humans and dogs, a second prediction frame set is obtained for the target object humans, and a second prediction frame set is obtained for the target object dogs.
[0074] Step 104 , performing weighted averaging processing on the prediction frames of the same target object in the second prediction frame set to obtain a detection frame set of the image data.
[0075] Here, the weighted averaging processing for the prediction boxes in the second prediction box set is usually the weighted averaging processing for the prediction boxes corresponding to the same target object. If the weighted averaging processing is performed on the prediction boxes of different target objects, it will be meaningless. The specific implementation of this point is usually achieved by setting a first threshold based on the intersection-union ratio.
[0076] Figure 2 is a flowchart showing a method of obtaining a detection frame set of a target object based on a second prediction frame set according to a specific embodiment. Figure 2 As shown, step 104 may further include the following steps 201 to 204.
[0077] Step 201: Select the prediction box with the highest confidence in the current second prediction box set as the candidate box.
[0078] Step 202, respectively calculate the intersection-over-union ratios of the candidate box and other prediction boxes in the second prediction box set.
[0079] The intersection over union (IOU) in this embodiment refers to the ratio of the area of the intersecting region to the area of the union region, which is simply the ratio of the intersection divided by the union.
[0080] By executing step 202, N-1 intersection-over-union ratios are obtained, where N is the total number of prediction frames in the second prediction frame set.
[0081] Step 203: If there is an IoU ratio greater than the first preset threshold value among the calculated IoU ratios, the prediction boxes corresponding to the IoU ratios greater than the first preset threshold value and the candidate boxes are weighted averaged to obtain the detection box of the target object and add it to the detection box set. This process ends.
[0082] The first preset threshold value is smaller than the second preset threshold value, and the second preset threshold value is the threshold value used by the NMS model.
[0083] In a specific implementation, the first preset threshold is set to be smaller than the second preset threshold because the relevant prediction boxes larger than the first preset threshold have been filtered out during NMS filtering.
[0084] For example, the second preset threshold value may be set to 0.65, and the first preset threshold value may be set to 0.45. There is no limitation on this in specific implementation, and it can be set according to actual needs.
[0085] In the embodiment of the present application, all prediction frames corresponding to the intersection-and-union ratios greater than the first preset threshold value are considered to be prediction frames corresponding to the same target object, and the prediction frames corresponding to each target object are obtained by setting the first preset threshold value and weighted averaging is performed.
[0086] When performing weighted averaging, the weight corresponding to the i-th prediction box is: the product of the confidence of the i-th prediction box and the corresponding intersection-and-union ratio; where the intersection-and-union ratio corresponding to the i-th prediction box is the intersection-and-union ratio of the i-th prediction box and the candidate box.
[0087] Step 204: If there is no IoU ratio greater than the first preset threshold value among the calculated IoU ratios, the candidate box is used as the detection box of the target object and added to the detection box set.
[0088] If the calculated intersection-over-union ratios are not greater than the first preset threshold value, it is considered that there are no other prediction boxes for the same target object as the candidate box in the second prediction box set, and they are directly used as the output result.
[0089] In the I / O ratios calculated in step 202, an I / O ratio greater than the first preset threshold is determined. For example, the second prediction frame set includes three prediction frames, namely: prediction frame 1, prediction frame 2, and prediction frame 3. Assuming that the confidence of prediction frame 1 is the largest, prediction frame 1 is the candidate frame, and the candidate frame is respectively calculated with the prediction frame 2 and the prediction frame 3 for I / O ratios 1 and I / O ratio 2 to obtain;
[0090] Assuming that both the intersection-over-union ratio 1 and the intersection-over-union ratio 2 are greater than the first preset threshold, it is considered that prediction box 1, prediction box 2 and prediction box 3 are prediction boxes of the same target object; prediction box 1, prediction box 2 and prediction box 3 are weighted averaged, and the new prediction box obtained is used as the final detection box of the target object.
[0091] Assuming that both the IoU 1 and the IoU 2 are not greater than the first preset threshold, that is, there is no IoU greater than the first preset threshold among the calculated IoUs, then the candidate box (prediction box 1) is directly used as the detection box of the target object.
[0092] At this point, the relevant processing of selecting a candidate box is completed.
[0093] From the above example, it can be seen that after all prediction frames in the second prediction frame set are processed at one time, the prediction frames in the second prediction frame set are considered to be prediction frames of the same target object.
[0094] Figure 3 is another flowchart of obtaining a detection frame set of a target object based on a second prediction frame set according to a specific embodiment. Figure 3 As shown, step 104 may further include the following steps 301 to 307 .
[0095] Step 301: Select the prediction box with the highest confidence in the current second prediction box set as the candidate box.
[0096] Step 302, calculating the intersection-over-union ratios of the candidate box and other prediction boxes in the second prediction box set respectively.
[0097] By executing step 302, N-1 intersection-over-union ratios are obtained, where N is the total number of prediction frames in the second prediction frame set.
[0098] Step 303: If there is an IoU greater than the first preset threshold value among the calculated IoUs, the prediction box corresponding to the IoU greater than the first preset threshold value and the candidate box are weighted averaged to obtain the detection box of the target object and add it to the detection box set.
[0099] The first preset threshold value is less than the second preset threshold value, and the second preset threshold value is the threshold value used by the NMS model;
[0100] In a specific implementation, the first preset threshold is set to be smaller than the second preset threshold because the relevant prediction boxes larger than the first preset threshold have been filtered out during NMS filtering.
[0101] For example, the second preset threshold value may be set to 0.65, and the first preset threshold value may be set to 0.45. There is no limitation on this in specific implementation, and it can be set according to actual needs.
[0102] In the embodiment of the present application, it is considered that all prediction boxes corresponding to the intersection-and-union ratios greater than the first preset threshold value are prediction boxes corresponding to the same target object, and the prediction boxes corresponding to each target object are obtained by setting the first preset threshold value and weighted averaging is performed.
[0103] When performing weighted averaging, the weight corresponding to the i-th prediction box is: the product of the confidence of the i-th prediction box and the corresponding intersection-and-union ratio; where the intersection-and-union ratio corresponding to the i-th prediction box is the intersection-and-union ratio of the i-th prediction box and the candidate box.
[0104] Step 304 : Delete the prediction box corresponding to the IoU ratio greater than the first preset threshold value and the candidate box from the second prediction box set. Execute step 307 .
[0105] These prediction boxes and candidate boxes are processed prediction boxes, and therefore, they are deleted from the second prediction box set to avoid repeated processing.
[0106] Step 305: If there is no IoU ratio greater than the first preset threshold value among the calculated IoU ratios, the candidate box is used as the detection box of the target object and added to the detection box set.
[0107] If the calculated intersection-over-union ratios are not greater than the first preset threshold value, it is considered that there are no other prediction boxes for the same target object as the candidate box in the second prediction box set, and they are directly used as the output result.
[0108] If there is only one prediction box in the second prediction box set, it is also considered that there is no intersection-over-union ratio greater than the first preset threshold value in the calculated intersection-over-union ratio.
[0109] Step 306: delete the candidate box from the second prediction box set.
[0110] In step 305, the candidate box has been processed and deleted to avoid repeated operations on the predicted box.
[0111] Step 307, determine whether the current second prediction box set is empty; if so, end this process; otherwise, execute step 301.
[0112] At this point, the processing of the entire second prediction frame set is completed, and the detection frame set is obtained.
[0113] This embodiment can handle the situation where there is one or more target objects in the second prediction frame set.
[0114] In the above two embodiments of the present application, when obtaining the detection frame set through the second prediction frame set, the processed prediction frame is deleted from the second prediction frame set, and the processed prediction frame is added to the detection frame set as the detection frame;
[0115] In the specific implementation, the processed prediction box can be marked, and then the unmarked prediction box can be traversed for processing, and the processed prediction box can be stored as the detection box;
[0116] The processed prediction frame can also be used to update the corresponding pre-processed detection frame and mark the prediction frame as the detection frame.
[0117] The specific method of this implementation is not limited in the embodiments of the present application.
[0118] Figure 2 and Figure 3 The corresponding disclosed embodiment gives in detail the weighted averaging process of the prediction box of a target object after filtering by the NMS model. During the weighted averaging process, the intersection-over-union ratio and confidence are taken into account, so that the detection box after the weighted averaging process has better accuracy and recall than the prediction box directly output by the NMS model.
[0119] Step 105: Perform object detection on the image data based on the detection frame set.
[0120] After using the NMS model to filter the prediction frame, this embodiment does not directly use the filtered prediction frame for target detection, but instead performs weighted averaging on the filtered prediction frames of the same target object to obtain the detection frame of the corresponding target object, and then performs target detection; this makes the obtained detection frame more precise and has better recall, thereby improving the accuracy of target detection.
[0121] Figure 4 is a flow chart of another target detection method according to an exemplary embodiment. Figure 4 As shown, the target detection method of the embodiment of the present disclosure mainly includes the following steps 401 to 408.
[0122] Step 401, acquiring image data.
[0123] The image data may be acquired in real time or may be data acquired in advance and stored, which is not limited in the embodiments of the present application. The stored data may be data stored in any form in any storage device.
[0124] Step 402: input the image data into a prediction model, and obtain a prediction box of a target object associated with the image data through the prediction model to form a first prediction box set.
[0125] Obtaining the first prediction box set in this step can be achieved through a prediction model. The prediction model can be a model established based on a neural network algorithm. However, in the specific implementation, there is no restriction on the method of obtaining the first prediction box set.
[0126] For each target object in the image data, multiple prediction boxes are predicted, as well as the confidence level corresponding to each prediction box, forming a first prediction box set.
[0127] Step 403: Filter the first prediction box set based on the NMS model to obtain a second prediction box set of the target object.
[0128] The NMS model is established based on the NMS algorithm. The process of filtering the prediction box by the NMS model can be:
[0129] The first step is to sort each prediction box in the first prediction set according to the confidence level, and select the prediction box with the highest confidence level as the candidate box;
[0130] The second step is to traverse the remaining prediction boxes. If a prediction box whose intersection-over-union ratio with the candidate box is greater than a second preset threshold value is found, the corresponding prediction box is deleted.
[0131] The third step is to select the prediction box with the highest confidence from the unprocessed prediction box as the candidate box again, and repeat the second step until all prediction boxes are processed.
[0132] The second prediction box set of the target object obtained after filtering by the NMS model will usually include multiple prediction boxes, but the number of prediction boxes corresponding to each target object will usually not exceed 5, as well as the confidence of each prediction box; this is just a conventional explanation. In the embodiments of the present application, there is no limit on the number of prediction boxes corresponding to each target object after filtering.
[0133] In the embodiment of the present application, there is no restriction on the filtering process of the prediction box based on the NMS model, and it can be implemented according to the relevant procedures.
[0134] When implementing the NMS model, it is very important to set the threshold value, which is not limited in the embodiment of the present application, and can be set to 0.65, for example.
[0135] If there are target objects of different categories, second prediction box sets are obtained for target objects of different categories respectively; and different second prediction boxes are processed respectively in subsequent processing.
[0136] If the target objects include humans and dogs, a second prediction frame set is obtained for the target object humans, and a prediction frame set is obtained for the target object dogs.
[0137] Step 404: Select the prediction box with the highest confidence in the current second prediction box set as the candidate box.
[0138] Step 405 , respectively calculating the intersection-over-union ratios of the candidate box and other prediction boxes in the second prediction box set.
[0139] Step 406: If there is an IoU greater than the first preset threshold value among the calculated IoUs, the prediction box corresponding to the IoU greater than the first preset threshold value and the candidate box are weighted averaged to obtain the detection box of the target object and add it to the detection box set.
[0140] The first preset threshold value is less than the second preset threshold value, and the second preset threshold value is the threshold value used by the NMS model;
[0141] When performing weighted averaging, the weight corresponding to the i-th prediction box is: the product of the confidence of the i-th prediction box and the corresponding intersection-and-union ratio; where the intersection-and-union ratio corresponding to the i-th prediction box is the intersection-and-union ratio of the i-th prediction box and the candidate box.
[0142] In specific implementation, it can be calculated by the following formula:
[0143]
[0144] Among them, w-box is the coordinate information of the detection box;
[0145]
[0146] n is the number of prediction frames participating in weighted averaging, that is, the total number of candidate frames and corresponding prediction frames whose intersection-over-union ratio is greater than the first preset threshold; C i Represents the confidence of the i-th prediction box; represents the intersection-over-union ratio of the i-th prediction box and the prediction box with the highest confidence; B i is the coordinate information of the i-th prediction box.
[0147] The n prediction boxes here include the candidate box and the corresponding prediction box whose intersection-over-union ratio is greater than a first preset threshold, the intersection-over-union ratio of the candidate box and the prediction box with the largest confidence, that is, the intersection-over-union ratio of the candidate box to itself, and is 1.
[0148] Step 407 : Delete the prediction boxes corresponding to the IoU ratios greater than the first preset threshold value and the candidate boxes from the second prediction box set. Execute step 410 .
[0149] Step 408: If there is no IoU ratio greater than the first preset threshold value among the calculated IoU ratios, the candidate box is used as the detection box of the target object and added to the detection box set.
[0150] If the calculated intersection-over-union ratios are not greater than the first preset threshold value, it is considered that there are no other prediction boxes for the same target object as the candidate box in the second prediction box set, and they are directly used as the output result.
[0151] If there is only one prediction box in the second prediction box set, it is also considered that there is no intersection-over-union ratio greater than the first preset threshold value in the calculated intersection-over-union ratio.
[0152] Step 409: delete the candidate box from the second prediction box set.
[0153] Step 410 , determine whether the current second prediction box set is empty, if so, execute 411 ; otherwise, execute step 404 .
[0154] Step 411, performing object detection on the image data based on the detection frame set.
[0155] After the prediction frame is filtered by the NMS model, the present embodiment does not directly use the filtered prediction frame for target detection, but performs weighted average processing on the filtered prediction frames of the same target object to obtain the detection frame of the corresponding target object, and considers the confidence of each prediction frame and the intersection-over-union ratio with the candidate frame when performing the weighted average processing, and then performs target detection; so that the obtained detection frame has better precision and recall, thereby improving the accuracy of target detection;
[0156] Furthermore, the solution provided in the embodiment of the present application is to perform weighting after the NMS model comes over, rather than before the NMS model comes over. This processing method does not introduce more computing overhead and ensures the speed of post-processing.
[0157] Take the image data as an example of a picture data in which the target object is a person, and perform target detection. That is, there is only one type of target object in the picture. The following is a detailed description of the target detection process for the image data in conjunction with the accompanying drawings.
[0158] The first step is to obtain image data.
[0159] The second step is to input the image data into a prediction model, and obtain a prediction box of a target object associated with the image data through the prediction model to form a first prediction box set.
[0160] Usually, multiple prediction boxes of the target object are obtained through the prediction model; assuming that the first prediction box set includes 5 prediction boxes, the first prediction box set is: {prediction box 1 (confidence 1), prediction box 2 (confidence 2), prediction box 3 (confidence 3), prediction box 4 (confidence 4), prediction box 5 (confidence 5)}; wherein the number of target objects can be one or more, and the prediction boxes of multiple target objects of the same type will be in one first prediction box set.
[0161] The third step is to filter the first prediction box set based on the NMS model to obtain the second prediction box set of the target object.
[0162] In this step, it is assumed that the confidence 1 of prediction box 1 is the largest, then prediction box 1 is the candidate box, and the candidate box is respectively calculated with prediction box 2, prediction box 3, prediction box 4 and prediction box 5 to obtain intersection and union ratio 1, intersection and union ratio 2, intersection and union ratio 3 and intersection and union ratio 4;
[0163] Assuming that both the intersection-and-union ratio 1 and the intersection-and-union ratio 2 are greater than the second preset threshold, and both the intersection-and-union ratio 3 and the intersection-and-union ratio 4 are not greater than the first preset threshold, prediction box 2 and prediction box 3 are filtered out when filtering through the NMS model, and the obtained second prediction box set is: {prediction box 1 (confidence 1), prediction box 4 (confidence 4), prediction box 5 (confidence 5)}.
[0164] Among them, when implementing the NMS model, a very important point is to set the threshold value. There is no restriction on this in the embodiment of the present application, for example, it can be set to 0.65.
[0165] Step 4: Select prediction box 1 with the highest confidence in the current second prediction box set as the candidate box.
[0166] Step 5: Calculate the intersection-and-union ratios of the candidate box and the prediction box 4 and the prediction box 5 respectively to obtain the intersection-and-union ratios 3 and 4.
[0167] In specific implementation, the intersection-and-union ratio calculated in the NMS model may be directly used, or the intersection-and-union ratio calculated again in this step may be used.
[0168] In the sixth step, assuming that the calculated intersection-over-union ratio 3 is greater than the first preset threshold value, and the intersection-over-union ratio 4 is not greater than the first preset threshold value, the prediction box 1 and the prediction box 4 are weighted averaged, and the obtained prediction box is used as the detection box of the target object and added to the detection box set.
[0169] The first preset threshold value set in the embodiment of the present application is smaller than the second preset threshold value, and the second preset threshold value is the threshold value used by the NMS model.
[0170] The prediction frame 1 and the prediction frame 4 are weighted averaged by the following formula:
[0171]
[0172] Among them, w1=C1×1=C1; w2=C2×I4; C1 is the confidence of prediction box 1, C2 is the confidence of prediction box 4; I4 is the intersection and union ratio of prediction box 1 and prediction box 4; B1 is the coordinate information of prediction box 1, and B2 is the coordinate information of prediction box 4.
[0173] Figure 5 FIG. 1 is a schematic diagram showing a comparison of a prediction frame and a detection frame corresponding to a target object according to an exemplary embodiment. Figure 5 As shown in FIG, the detection frame is obtained by performing weighted average processing on the prediction frame 1 and the prediction frame 4; wherein the prediction frame 1 is Figure 5 The prediction box 4 is represented by a thick solid box in Figure 5 The detection box is represented by a thin solid frame in Figure 5 Use in Wireframe representation; the detection frame can successfully match the GT frame; where the GT frame is Figure 5 Use in Wireframe representation so there are no false detections.
[0174] Step 7: Delete prediction box 1 and prediction box 4 from the second prediction box set.
[0175] In the eighth step, it is determined that the current second prediction box set is not empty, and it is determined again that the prediction box with the highest confidence in the second prediction box set is prediction box 5, and there are no other prediction boxes. It is determined that there is no prediction box whose intersection-over-union ratio is greater than the first preset threshold value, and prediction box 5 is directly added to the detection box set as a detection box, and prediction box 5 is deleted from the second prediction box set.
[0176] Based on the above processing, it can be seen that the prediction box 1 and the prediction box 4 here are the prediction boxes of one target object, and the prediction box 5 is the prediction box of another target object.
[0177] At this point, the final set of detection frames is generated, which is the set of detection frames for the image data.
[0178] Step 9: Perform target detection on image data based on the detection frame set.
[0179] The traditional NMS model selects the box with the highest confidence each time, which may not be accurately positioned, and the redundant box is also likely to be accurately positioned. Therefore, directly using the prediction box obtained by filtering the NMS model as the detection result may not have a single prediction box that is accurate. In the disclosed embodiment, the prediction box after filtering by the NMS model is judged by a first preset threshold, and the prediction box that meets the conditions is weighted averaged. The weighted average processing process comprehensively considers the intersection and union ratio and confidence of all candidate boxes. The obtained weighted average result box makes each target correspond to only one detection box, and the detection box can be matched with the real box. In this way, even if the more accurate prediction box is suppressed after filtering by the NMS model, or the prediction boxes left after filtering are inaccurate, a relatively accurate detection box can be obtained. Therefore, the accuracy and recall of the detection box of the target object obtained are better, thereby making the accuracy of target detection higher.
[0180] In specific applications, target object identification and tracking are performed based on the target detection results. This is not limited in the embodiments of the present application, and the target detection results obtained in the embodiments of the present application can be used according to actual application scenarios.
[0181] Figure 6 is a schematic diagram of a target detection device according to an exemplary embodiment, such as Figure 6 As shown, the target detection device mainly includes:
[0182] An acquisition unit 601 is configured to acquire image data;
[0183] The prediction unit 602 is configured to input the image data into a prediction model, and obtain a prediction box of a target object associated with the image data through the prediction model to form a first prediction box set;
[0184] A filtering unit 603 is configured to filter the first prediction frame set based on a non-maximum suppression (NMS) model to obtain a second prediction frame set for the target object;
[0185] The processing unit 604 is configured to perform weighted averaging processing on the prediction frames of the same target object in the second prediction frame set to obtain a detection frame set of the image data;
[0186] The detection unit 605 is configured to perform target detection on the image data based on the detection frame set.
[0187] In some embodiments,
[0188] The processing unit 604 is configured to perform weighted averaging processing on the prediction frames of the same target object in the second prediction frame set, and when obtaining the detection frame set of the image data, select the prediction frame with the highest confidence in the current second prediction frame set as the candidate frame; respectively calculate the intersection and union ratios of the candidate frame and other prediction frames in the second prediction frame set; if there is an intersection and union ratio greater than a first preset threshold value among the calculated intersection and union ratios, the prediction frame corresponding to the intersection and union ratio greater than the first preset threshold value and the candidate frame are weighted averaged to obtain the detection frame of the target object and add it to the detection frame set; wherein the first preset threshold threshold is less than the second preset threshold threshold, and the second preset threshold threshold is the threshold threshold used by the NMS model.
[0189] In some embodiments,
[0190] The processing unit 604 is configured to execute, if there is no IoU ratio greater than a first preset threshold value in the calculated IoU ratios, taking the candidate frame as the detection frame of the target object and adding it to the detection frame set.
[0191] In some embodiments,
[0192] The processing unit 604 is configured to perform weighted averaging processing on the prediction frame corresponding to the intersection-and-union ratio greater than the first preset threshold value and the candidate frame, and then delete the prediction frame corresponding to the intersection-and-union ratio greater than the first preset threshold value and the candidate frame from the second prediction frame set; or, if the calculated intersection-and-union ratio does not contain an intersection-and-union ratio greater than the first preset threshold value, delete the candidate frame from the second prediction frame set.
[0193] In some embodiments,
[0194] The processing unit 604 is configured to determine whether the current second prediction frame set is empty; if it is not empty, perform the step of selecting the prediction frame with the highest confidence in the current second prediction frame set as the candidate frame and subsequent steps until the second prediction frame set is empty.
[0195] In some embodiments,
[0196] The processing unit 604 is configured to perform weighted averaging processing, and the weight corresponding to the i-th prediction box is: the product of the confidence of the i-th prediction box and the corresponding intersection-and-union ratio; wherein the intersection-and-union ratio corresponding to the i-th prediction box is the intersection-and-union ratio of the i-th prediction box and the candidate box.
[0197] In some embodiments,
[0198] If there are target objects of different categories, the second prediction box sets are obtained for target objects of different categories respectively.
[0199] Regarding the target detection device in each of the above embodiments, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the target detection method, and will not be elaborated here.
[0200] It should be noted that: the above embodiment only illustrates the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0201] Figure 7 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. In some embodiments, the electronic device is a server. The electronic device 700 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 701 and one or more memories 702, wherein at least one program code is stored in the memory 702, and the at least one program code is loaded and executed by the processor 701 to implement the target detection method provided by each of the above embodiments. Of course, the electronic device 700 may also have components such as image acquisition components, wired or wireless network interfaces, keyboards, and input and output interfaces for input and output, and the electronic device 700 may also include other components for implementing device functions, which will not be repeated here.
[0202] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction. The at least one instruction can be executed by a processor in a computer device to complete the target detection method in the above embodiment.
[0203] Optionally, the above-mentioned computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may include ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device, etc.
[0204] In an exemplary embodiment, a computer program product is also provided, including one or more instructions, which can be executed by a processor of a computer device to complete the target detection method provided by each of the above embodiments.
[0205] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.
Claims
1. A target detection method, characterized in that: The method comprises: Get image data; Inputting the image data into a prediction model, obtaining a prediction frame of a target object associated with the image data through the prediction model to form a first prediction frame set; Filter the first prediction box set based on a non-maximum suppression (NMS) model to obtain a second prediction box set for the target object; Performing weighted averaging processing on the prediction frames of the same target object in the second prediction frame set to obtain a detection frame set of the image data; Perform object detection on the image data based on the detection frame set.
2. The method according to claim 1, characterized in that: The step of performing weighted averaging processing on the prediction frames of the same target object in the second prediction frame set to obtain the detection frame set of the image data includes: Select the prediction box with the highest confidence in the current second prediction box set as the candidate box; Calculating the intersection-over-union ratios of the candidate box and other prediction boxes in the second prediction box set respectively; If there is an intersection-and-union ratio greater than a first preset threshold value among the calculated intersection-and-union ratios, the prediction box corresponding to the intersection-and-union ratio greater than the first preset threshold value and the candidate box are weighted averaged to obtain the detection box of the target object and add it to the detection box set; wherein the first preset threshold value is less than the second preset threshold value, and the second preset threshold value is the threshold value used by the NMS model.
3. The method according to claim 2, characterized in that The method further comprises: If there is no intersection-over-union ratio greater than the first preset threshold value among the calculated intersection-over-union ratios, the candidate frame is used as the detection frame of the target object and added to the detection frame set.
4. The method according to claim 3, characterized in that: The method further comprises: If the prediction frame corresponding to the intersection-and-union ratio greater than the first preset threshold value and the candidate frame are subjected to weighted averaging processing, the prediction frame corresponding to the intersection-and-union ratio greater than the first preset threshold value and the candidate frame are deleted from the second prediction frame set; or, If there is no intersection-over-union ratio greater than a first preset threshold value among the calculated intersection-over-union ratios, the candidate frame is deleted from the second prediction frame set.
5. The method according to claim 4, characterized in that The method further comprises: Determine whether the second prediction frame set is currently empty; If it is not empty, the step of selecting the prediction box with the highest confidence in the current second prediction box set as the candidate box and subsequent steps is performed until the second prediction box set is empty.
6. The method according to claim 2, characterized in that When performing the weighted average processing, the weight corresponding to the i-th prediction box is: the product of the confidence of the i-th prediction box and the corresponding intersection-and-union ratio; wherein the intersection-and-union ratio corresponding to the i-th prediction box is the intersection-and-union ratio of the i-th prediction box and the candidate box.
7. The method according to any one of claims 1 to 6, characterized in that: If there are target objects of different categories, the second prediction box sets are obtained for target objects of different categories respectively.
8. A target detection device, characterized in that: The device comprises: An acquisition unit, configured to acquire image data; A prediction unit is configured to input the image data into a prediction model, and obtain a prediction box of a target object associated with the image data through the prediction model to form a first prediction box set; A filtering unit is configured to filter the first prediction box set based on a non-maximum suppression (NMS) model to obtain a second prediction box set for the target object; a processing unit configured to perform weighted averaging processing on the prediction frames of the same target object in the second prediction frame set to obtain a detection frame set of the image data; A detection unit is configured to perform target detection on the image data based on the detection frame set.
9. An electronic device, characterized in that: include: processor; a memory for storing executable instructions for the processor; The processor is configured to execute the executable instructions to implement the target detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: When at least one instruction in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to implement the target detection method as described in any one of claims 1 to 7.