Target detection method and device based on integrated model

Abstract

The invention provides a target detection method and device based on an integrated model, and belongs to the technical field of deep learning, and the method comprises the steps: inputting a to-be-detected image into a differentiated integrated model for detection, and obtaining a plurality of detection results; wherein the differentiated integration model is obtained by performing parallel training on a plurality of sub-models by taking integration loss as a training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and a gradient included angle cosine value between the loss functions of the plurality of sub-models; the target detection loss comprises at least one of positioning loss and classification loss; and judging the plurality of detection results to obtain a judgment result. According to the method provided by the invention, the detection performance of the whole model is improved, and good robustness can be shown in the face of adversarial attacks, so that the accuracy of a target detection result is improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a target detection method and device based on an integrated model. Background Art

[0002] Currently, object detection technologies have been widely used in many fields such as face recognition and autonomous driving. However, these technologies are vulnerable to adversarial attacks, in which attackers may deceive detection systems by attaching adversarial patches to physical objects. These vulnerabilities have raised concerns about the reliability and security of critical applications.

[0003] In the relevant technical field, there are many exploitable vulnerabilities in the field of target detection. The existing target detection models have low adversarial robustness and cannot effectively resist adversarial ensemble attacks. Moreover, in the adversarial training process, it is particularly difficult to balance the robustness and generalization of the model, which leads to unstable model performance and further reduces the accuracy of target detection. Summary of the invention

[0004] In view of the problems existing in the prior art, an embodiment of the present invention provides a target detection method and device based on an integrated model.

[0005] In a first aspect, the present invention provides a target detection method based on an integrated model, comprising: The image to be detected is input into a differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by training multiple sub-models in parallel with an integrated loss as a training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value of the gradient angle between the loss functions of the multiple sub-models; the target detection loss includes at least one of a positioning loss and a classification loss; A decision is made on the multiple detection results to obtain a decision result.

[0006] Optionally, the detection loss is determined based on the positioning loss and the classification loss; The integrated loss is obtained by the following steps: The multiple sub-models are trained in parallel, and for each sub-model, the detection loss of the sub-model is calculated according to the input and output of the sub-model; Obtaining a cosine loss of a gradient angle between the multiple sub-models according to a gradient of the target detection loss with respect to the input; The cosine loss and the target detection loss of each sub-model are weightedly calculated to obtain the integrated loss.

[0007] Optionally, obtaining the cosine loss of the gradient angle between the multiple sub-models according to the gradient of the target detection loss with respect to the input includes: ; in, is the cosine loss, For the i The object detection loss of the sub-model is about the input x The gradient of For the j The object detection loss of the sub-model is about the input x The gradient of N is the number of sub-models.

[0008] Optionally, the differentiated integrated model is obtained by applying an integrated loss function and performing back-propagation training on the multiple sub-models.

[0009] Optionally, the multiple sub-models adopt exactly the same architecture, and the inputs of the multiple sub-models are the same.

[0010] Optionally, the adjudicating the multiple detection results to obtain the adjudication result comprises: In a case where a first detection box in the multiple detection results is detected by a first number of sub-models, deleting the first detection box from the multiple detection results; in a case where a second detection box in the multiple detection results is detected by a second number of sub-models, fusing the second detection boxes detected by the second number of sub-models to obtain a plurality of filtered detection results; wherein the first number does not exceed half of the number of sub-models, and the second number is greater than the first number; The multiple screened test results are combined to obtain the decision result.

[0011] Optionally, different detection results correspond to different two-dimensional matrices, the first detection box belongs to a subvector of a first two-dimensional matrix, and the second detection box belongs to a subvector of a second two-dimensional matrix; Deleting the first detection frame from the multiple detection results includes: The first detection frame in the first two-dimensional matrix is ​​reset to zero.

[0012] Optionally, there are overlapping detection frames in the multiple detection results, and fusing the second detection frames detected by the second number of sub-models to obtain the multiple filtered detection results includes: Get screening test results by any of the following: If there are overlapping detection frames in the multiple detection results, any two overlapping detection frames are recorded as B 1and B 2 ; Use the following formula to B 1 and B 2 Fusion is performed to obtain new detection results: ; in, For the new test result, for B 1 The area of for B 2 area; Alternatively, different detection results correspond to different two-dimensional matrices, the first detection box belongs to a sub-vector of the first two-dimensional matrix, and the second detection box belongs to a sub-vector of the second two-dimensional matrix; any one of a second number of second detection boxes is retained from the multiple detection results, and the other second detection boxes other than the any one second detection box are reset to zero to obtain the multiple filtered detection results.

[0013] In a second aspect, the present invention further provides a target detection device based on an integrated model, comprising: An integrated detection module, used for inputting the image to be identified into a differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by back-propagation training of multiple sub-models with an integrated loss as a training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value of the gradient angle between the loss functions of the multiple sub-models; the target detection loss includes at least one of a positioning loss and a classification loss; The decision module is used to make a decision on the multiple detection results to obtain a decision result.

[0014] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the above-described integrated model-based target detection methods.

[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described integrated model-based target detection methods.

[0016] In a fifth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned integrated model-based target detection methods.

[0017] The target detection device based on the integrated model provided by the present invention detects the input image through the differentiated integrated model to obtain multiple detection results, and judges the multiple detection results to obtain the judgement result. Compared with the single model, the detection performance of the whole model is improved; since the differentiated integrated model is obtained by back-propagation training of multiple sub-models with the integrated loss as the training loss function, and the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine loss of the gradient angle between the multiple sub-models, the differentiated integrated model can show good robustness in the face of integrated attacks, thereby improving the accuracy of the target detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the target detection method based on the integrated model provided by the present invention.

[0020] Figure 2 This is the second flow chart of the target detection method based on the integrated model provided by the present invention.

[0021] Figure 3 This is the third flow chart of the target detection method based on the integrated model provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the target detection device based on the integrated model provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Combine the following Figure 1-Figure 4 The object detection method and device based on integrated model of the present invention are described.

[0026] Figure 1is one of the flow charts of the target detection method based on the integrated model provided by the present invention, such as Figure 1 As shown, the target detection method based on the integrated model includes the following steps: Step 110: input the image to be detected into the differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by training multiple sub-models in parallel with the integrated loss as the training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value of the gradient angle between the loss functions of the multiple sub-models; the target detection loss includes at least one of the positioning loss and the classification loss.

[0027] Optionally, the image to be detected includes an image or video frame captured in real time, and also includes an image stored in a database.

[0028] Optionally, the initial model of the differentiated integrated model includes multiple sub-models, and the network architecture of each sub-model is set to be exactly the same.

[0029] Optionally, the detection result includes the coordinates, category and confidence level of the object to be detected.

[0030] Optionally, the integrated loss is a total loss function in the model training process, wherein the integrated loss includes the integrated loss obtained by accumulating and summing the target detection losses corresponding to each sub-model and the above-mentioned cosine loss.

[0031] Optionally, for the ensemble model F Included N The sub-models are trained to obtain The positioning loss and classification loss .

[0032] Optionally, for the same loss function, the target detection loss of different sub-models is compared with the common input x The gradient information between them is used to calculate the cosine loss value of the gradient angle corresponding to each sub-model.

[0033] Optionally, the integration loss corresponding to the multiple sub-models is determined by the cosine loss and the target detection loss of each sub-model, and each sub-model is iteratively trained with the integration loss, and the corresponding differentiated integration model is optimized after the target number of iterations.

[0034] for example, n The integrated loss of each sub-model is the positioning loss ; can be based on n The localization loss of each sub-model is related to the common input x Calculate the gradient between each sub-model .

[0035] for example, n The integrated loss of each sub-model includes the positioning loss and classification loss , expressed as , ; can be based on n The localization loss of each sub-model is related to the common input x Calculate the gradient between each sub-model .

[0036] Optionally, the iterative training method includes gradient descent method, Newton method, conjugate gradient method and attenuated least squares method, etc.

[0037] Step 120: adjudicate multiple detection results to obtain a decision result.

[0038] Optionally, the image to be detected is simultaneously input to N After the sub-model is tested, the output N The detection results (coordinates, categories and confidence levels of the objects to be detected) are screened and integrated according to the preset decision algorithm to obtain the decision result.

[0039] Optionally, a decision can be made on multiple detection results by deleting a small amount of abnormal information in the detection results. For example, if there is a detection box in multiple detection results that is only detected by a specified number of sub-models, the detection box can be deleted to obtain a decision result. At the same time, multiple overlapping information in the detection results can be merged. For example, if there are overlapping detection boxes in multiple detection results, the overlapping detection boxes can be fused to obtain a unique decision result.

[0040] Optionally, the image to be detected x Input to the differentiated integrated model N The detection is performed in the trained sub-models, and the integrated loss of the differentiated integrated model is obtained by weighted combination of the target detection loss and cosine loss corresponding to each sub-model; N The trained sub-models are used for images x Perform target detection and output N Test results; statistics N If the number of overlapping frames corresponding to the detection results is lower than the first threshold, N If the number of overlapping frames is higher than the second threshold, the overlapping frames are removed from the detection results. NThe overlapping frames are fused in the detection results, only one overlapping frame is retained, and finally the detection results that have been eliminated and fused are merged to obtain a unique judgment result; wherein, the values ​​of the first threshold and the second threshold can be set according to user needs, and the second threshold is greater than the first threshold. The target detection method based on the integrated model provided by the embodiment of the present invention detects the input image through a differentiated integrated model to obtain multiple detection results, and judges the multiple detection results to obtain a judgment result, which improves the detection performance of the entire model compared to a single model; since the differentiated integrated model is obtained by training multiple sub-models with the integrated loss as the training loss function, and the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine loss of the gradient angle between the multiple sub-models, the robustness of the differentiated integrated model to cope with integrated attacks is improved, thereby improving the accuracy of the target detection results.

[0041] Furthermore, the target detection loss is determined based on the positioning loss and the classification loss; the integrated loss in step 110 is obtained by the following steps: multiple sub-models are trained in parallel, and for each sub-model, the target detection loss of the sub-model is calculated according to the input and output of the sub-model; according to the gradient of the target detection loss with respect to the input, the cosine loss of the gradient angle between the multiple sub-models is obtained; the cosine loss and the target detection loss of each sub-model are weighted and calculated to obtain the integrated loss.

[0042] Optionally, when the target detection loss includes one loss function, the corresponding cosine loss is the cosine loss value of the loss function solved by the cosine theorem, and when the target detection loss includes two loss functions, the corresponding cosine loss is the cumulative value of the cosine losses solved by the cosine theorem of the two loss functions. Optionally, the integrated loss is expressed by the following formula: ; in, is the integrated loss, L n is the target detection loss corresponding to each sub-model, N is the number of sub-models.

[0043] Optionally, the target detection loss corresponding to each sub-model includes two parts: positioning loss and classification loss. x Input each sub-model at the same time, and the output is , and then calculate the positioning loss corresponding to the output according to the positioning loss function , according to the classification loss function, calculate the classification loss corresponding to the output ; The target detection loss can be expressed as follows: ; ; in, Logo n The target detection loss corresponding to the sub-model is: For images .

[0044] Optionally, the gradient differentiation loss of the ensemble model It can be expressed by the following formula: .

[0045] Optionally, before converting the image x Before inputting each sub-model at the same time, the image needs to be x Preprocessing is performed. Specifically, the preprocessed image is obtained by randomly cropping, rotating, and other common data enhancement methods on the input image. It should be noted that the image x The size of is adjusted to the same as the input size of each sub-model.

[0046] Optionally, obtain the ensemble loss by: (1) After determining the target detection loss corresponding to each model, the target detection loss of each model and the image x The gradient between the models is calculated according to the following formula .

[0047] Optionally, the gradients between models are calculated by calling the gradient calculation function from the database; for example, calling the built-in function in the pytorch library to calculate the gradients between models ; The specific calculation method is as follows: ; in, torch.autograd.grad Built-in functions for calculating gradients in the pytorch library.

[0048] (2) Using the gradients between sub-models The cosine loss of the angle between the gradients of different sub-models is calculated by the cosine theorem , .

[0049] (3) The above cosine loss value and the target detection loss corresponding to each sub-model are calculated by the following formula to obtain the corresponding integrated loss: ; in, is the coefficient of differentiation loss, The value can be set according to user needs, for example, The acceptable value is 0.001.

[0050] Optionally, a differentiated integrated model is trained by the following steps: (1) Obtain a dataset. The dataset consists of 17,125 RGB color images of different sizes, with a total of 20 categories.

[0051] (2) Preprocess the data set; first preprocess the data set by random cropping and rotation and other data enhancement methods, and then set the image size to 416*416 according to the architecture requirements of the YOLOv3 model to obtain the input image x , the corresponding label is .

[0052] (3) Initialize the integrated model; construct three YOLOv3 models with the same architecture and different network weight settings, and merge them into an integrated model. The integrated model shares the same set of inputs, and the target detection loss is equal to the sum of the losses of all sub-models, as shown in the following formula: .

[0053] (4) According to the target detection loss and the input dataset ( x , ) According to the following formula, the cosine value of the gradient angle between the three YOLOv3 models under the same loss (including the cosine value of the gradient angle corresponding to the positioning loss) is calculated respectively. The cosine of the angle between the gradient and the classification loss ): ; .

[0054] (5) Using the following formula based on the above , And the target detection loss calculates the final integrated loss: .

[0055] (6) Use the gradient descent method to train multiple sub-models based on the above integrated loss; specifically, according to the input image x Calculate the partial derivative of each parameter in the three YOLOv3 models with respect to the integrated loss, obtain the gradient information of each YOLOv3 model parameter, and then use the back propagation optimization method to update the network parameters to minimize the integrated loss value. When each YOLOv3 model reaches the maximum number of iterations or the integrated loss value is less than the preset threshold, a differentiated integrated model is obtained. .

[0056] Figure 2 This is the second flow chart of the target detection method based on the integrated model provided by the present invention. Figure 2In the illustrated embodiment, the integrated model is first initialized, the target detection loss of each sub-model is calculated for the input and output of each sub-model, and the sub-model loss gradient information is calculated based on the target detection loss and input of the sub-model, and the cosine loss corresponding to the gradient angle between the sub-models is further calculated according to the cosine theorem, and then the total loss (integrated loss) is determined based on the target detection loss and the cosine loss to iteratively train the integrated model to obtain a differentiated integrated model; then, the test image is input into the differentiated integrated model to obtain multiple outputs (corresponding to output 1 to output n, respectively), and the multiple outputs are input into the multi-model output arbiter for arbitration to obtain the final output.

[0057] The target detection method based on integrated model provided by the embodiment of the present invention calculates the cosine loss of the gradient angle between each sub-model through the target detection loss between the input and output of each sub-model, and weights the cosine loss and the target detection loss of each sub-model to obtain the integrated loss. Finally, differentiated training of multiple sub-models is realized according to the integrated loss. When facing adversarial samples generated based on gradient attacks, the sub-models that are not attacked can well ignore the influence of adversarial disturbances, weaken the mobility of adversarial samples in differentiated sub-models, and thus improve the accuracy of the model output results.

[0058] Furthermore, according to the gradient of the target detection loss with respect to the input, the cosine loss of the gradient angle between multiple sub-models is obtained, including: ; in, is the cosine loss, For the i The object detection loss of the sub-model is about the input x The gradient of For the j The object detection loss of the sub-model is about the input x The gradient of N is the number of sub-models.

[0059] The integrated model-based target detection method provided by an embodiment of the present invention can quickly obtain the cosine loss of the gradient angle between multiple sub-models through the above-mentioned cosine loss calculation method, thereby improving the target detection operation efficiency.

[0060] Furthermore, the differentiated ensemble model is obtained by applying the ensemble loss function and back-propagation training on multiple sub-models.

[0061] Optionally, the above integrated model can be Back-propagation training is performed to obtain a usable differentiated integrated model when the model is iterated to the maximum number of times or the model converges.

[0062] Optionally, each sub-model increases the gradient differentiation between the sub-models by reducing the cosine value of the angle in the above formula during back propagation, and finally generates a differentiated integrated model.

[0063] The target detection method based on the integrated model provided by the embodiment of the present invention is obtained by back-propagating multiple sub-models through the differentiated integrated model through the integrated loss, thereby improving the robustness of model detection.

[0064] Furthermore, multiple sub-models use exactly the same architecture, and the inputs of multiple sub-models are the same.

[0065] Optionally, the input of each sub-model remains the same; for example, multiple sub-models can be set as 3 YOLOv3 network models, and the network weights of each model are set differently; during the training process, the input image x is simultaneously input into the three YOLOv3 network models for parallel training.

[0066] Optionally, the sub-model can be a convolutional neural network, a deep neural network, or other neural networks.

[0067] The target detection method based on integrated model provided by the embodiment of the present invention can reduce the problem of consistent adversarial defects exhibited by neural networks with heavy target detection tasks and improve the robustness of model training by setting multiple sub-models to adopt exactly the same architecture and the same inputs of multiple sub-models.

[0068] Furthermore, in step 120, a plurality of detection results are judged, and the judgment result obtained includes: when a first detection box in the plurality of detection results is detected by a first number of sub-models, deleting the first detection box from the plurality of detection results; when a second detection box in the plurality of detection results is detected by a second number of sub-models, fusing the second detection boxes detected by the second number of sub-models to obtain a plurality of screened detection results; wherein the first number does not exceed half of the number of sub-models, and the second number is greater than the first number; and merging the plurality of screened detection results to obtain a judgment result.

[0069] Optionally, the first number can be set according to user needs; for example, the first number is no more than half of the number of all sub-models.

[0070] Optionally, place the image x The input is fed into three YOLOv3 networks with the same architecture but different network weight settings, and three different detection results A, B, and C are obtained respectively. Among them, the detection box a is only marked in the detection result A. There is no detection box a in the detection results B and C, indicating that the detection box a is an abnormal detection content. In this case, the detection box a should be deleted from the detection result A to obtain the detection result after removing a few isolated detection boxes. .

[0071] Optionally, according to the above test results And the original detection results B and C, if there are overlapping detection frames b in both detection results B and C, merge the overlapping detection frames b in detection results B and C into one detection frame , at this time, the new test result and No longer contains the detection box b, and finally , , and to merge.

[0072] Optionally, according to the above test results And the original detection results B and C, if there are overlapping detection frames b in both detection results B and C, the detection frame b of detection result B can also be removed, and the detection frame b in detection result C is retained. , Merge with C; or remove the detection box b in the detection result C, retain the detection box b in the detection result B, and finally , B and to merge.

[0073] Optionally, merging multiple detection results may be performed by rearranging the numerical matrices corresponding to the multiple detection results to obtain corresponding decision results.

[0074] Optionally, multiple test results may be merged, or the multiple test results may be filled into a list according to a preset arrangement order, and the filled list is the decision result.

[0075] The integrated model-based target detection method provided by an embodiment of the present invention improves the accuracy of the detection results by deleting a specific detection frame from multiple detection results on the basis that the detection frame is only detected by a first number of sub-models, and fusing multiple overlapping detection frames from multiple detection results to obtain a final judgment result.

[0076] Furthermore, different detection results correspond to different two-dimensional matrices, the first detection box belongs to a sub-vector of the first two-dimensional matrix, and the second detection box belongs to a sub-vector of the second two-dimensional matrix; deleting the first detection box from multiple detection results includes: resetting the first detection box in the first two-dimensional matrix to zero.

[0077] Optionally, the image to be detected x The differentiated integrated model outputs N detection results, each of which is represented in the form of a two-dimensional matrix, where the row data of the matrix can include the center point coordinates, width, height, classification confidence and object category of the detection box corresponding to the target object, and the column data of the matrix is ​​the number of targets to be detected.

[0078] Optionally, the image to be detected x The differentiated integrated model outputs three detection results, namely two-dimensional matrices A, B and C. Among them, if the first detection box (column vector a) exists in the two-dimensional matrix A, and the column vector a does not exist in the two-dimensional matrices B and C, it means that the column vector a is an abnormal detection content, and the column vector a in the two-dimensional matrix A should be converted into a zero vector to obtain a two-dimensional matrix A1.

[0079] The target detection method based on the integrated model provided by the embodiment of the present invention eliminates abnormal detection content from the detection results by resetting the first detection box in the first two-dimensional matrix to zero. The calculation method is simple, the operability is strong, and the efficiency of detecting result screening is improved.

[0080] Further, the second detection frames detected by the second number of sub-models are fused to obtain multiple filtered detection results, including: obtaining the filtered detection results by any of the following items: The first implementation method is: if there are overlapping detection frames in the multiple detection results, any two overlapping detection frames are respectively recorded as B 1 and B 2 ; Use the following formula to B 1 and B 2. Fusion to obtain new detection results: ; in, For the new test result, for B The area of ​​1, for B 2; the second implementable manner is: different detection results correspond to different two-dimensional matrices, the first detection frame belongs to a subvector of the first two-dimensional matrix, and the second detection frame belongs to a subvector of the second two-dimensional matrix; any one of the second number of second detection frames is retained from the multiple detection results, and the other second detection frames other than any one second detection frame are reset to zero, so as to obtain multiple filtered detection results.

[0081] For the first possible implementation method, the decision result is obtained through the above decision method. The steps are: (a) Assume that the integrated model is composed of N sub-models, corresponding to N Group Output List ; (b) N Group outputs consolidated into one master list ; (c) Traversal The detection box in the statistics of the current detection box The number of repeated boxes (IoU greater than the threshold) in , if the number is less than , then the box is determined to be the box predicted by the minority sub-model and is discarded according to the voting principle; (d) Sort by confidence from high to low and initialize the final output list ,Will The detection box with the highest confidence is added ; (e) Traverse backward , remove the current frame , if in There is a box in and If it is a repeating frame, and Fusion is performed and the fused frame is replaced If there is no satisfying condition, Join middle; (f) Repeat step (e) until Traversal completed; (g) The final output result is .

[0082] In this embodiment, define k Output list of submodels Pass m Detection boxes The format is as follows: ; ; in is the center coordinate of the current detection frame, and are the width and height of the detection box, respectively, and and They are the confidence score and category corresponding to the current detection box respectively.

[0083] The formula for deleting the detection boxes predicted by the minority models in the above step (c) can be expressed as: ; in, Indicates k The sub-model i The number of repeated boxes between the box and other detection boxes of all sub-models, where is a function that calculates the intersection-and-union ratio between two boxes. is an indicator function, when the condition x Its value is 1 if it is true, otherwise it is 0; Represents the threshold used to determine whether two boxes are overlapping boxes; if Less than , then the current Boxes predicted by the minority model are removed from the output list.

[0084] Optionally, after eliminating the boxes predicted by a few sub-models, the remaining overlapping boxes are weighted fused, with the size of the weight determined by the area of ​​the box; introducing the concept of area weight can give the weight a certain degree of dynamics, making it more flexible in most scenarios.

[0085] Optionally, in the above step (e), the list can be judged by weighting O Check if there is a box that overlaps with the current box.

[0086] For example, suppose two detection boxes and It is an overlapping frame, with the same format as , at this time they need to be merged into a new detection frame ; and The areas are expressed as and ; and The fusion weights are expressed as: ; ; Then get the fusion frame The formula is as follows: ; ; ; ; ; .

[0087] in, W p For the detection frame The fusion weight of W q For the detection frame The fusion weight of is the center coordinate of the fusion frame, and are the width and height of the fusion box, respectively, and and are the confidence scores and categories corresponding to the fusion frames respectively; through the above detection frame fusion method, the overlapping detection frames in multiple detection results can be merged in sequence to finally obtain a valid detection frame.

[0088] Optionally, if the list O There are multiple overlapping detection boxes in the list. O The detection box with the highest confidence B 0 and overlapping detection boxes B 1 is integrated to obtain B new1 ; If the list O Existence and B new1 Overlapping detection boxes B 2. Continue to B new1 and B 2 is fused to obtain B new2 ; and so on, until the list O Until there is no overlapping box with the detection box obtained by the previous fusion.

[0089] Optionally, during the fusion process, take the above B 1 and B The maximum confidence of 2 and the category of the maximum confidence are used as the fused new box B new The classification confidence and the category of the object in the box can be used to obtain the final output result based on the maximum confidence and corresponding category among multiple detection results, further improving the accuracy of the detection results.

[0090] For the second possible implementation method, the image to be detected is x The differentiated integrated model outputs three detection results, namely two-dimensional matrices A, B and C. Among them, if the first detection box (column vector a) exists in the two-dimensional matrix A, and the column vector a does not exist in the two-dimensional matrices B and C, it indicates that the column vector a is an abnormal detection content, and the column vector a in the two-dimensional matrix A should be converted into a zero vector to obtain the two-dimensional matrix A1; if the second detection box (column vector b) exists in the two-dimensional matrices A1, B and C, only the column vector b in the two-dimensional matrix B can be retained, and the column vectors b in the two-dimensional matrices A1 and C can be converted into zero vectors to obtain the two-dimensional matrices A2, B and C1.

[0091] Finally, the two-dimensional matrices A2, B and C1 are filled into the list in a preset order, and the zero vectors in the list are removed to obtain the corresponding judgment result. Optionally, the detection results output by each sub-model include the detection frame corresponding to the object to be detected (including the width, height and coordinates of the center point in the frame), the category of the object in the frame and the classification confidence of the detection frame.

[0092] The target detection method based on an integrated model provided by an embodiment of the present invention can retain the detection content with the highest confidence at each screening through the above-mentioned detection frame fusion method, thereby improving the accuracy of the decision; and by retaining any one of the second number of second detection frames from multiple detection results and resetting the other second detection frames other than any second detection frame to zero, redundant detection content is eliminated, the calculation method is simple, and the efficiency of detection result screening is improved.

[0093] Figure 3 This is the third flow chart of the target detection method based on the integrated model provided by the present invention. Figure 3 In the embodiment shown, in the training phase, the data sets are input into values n models with the same architecture (corresponding to F 1- F n ), the target detection loss corresponding to each model is calculated based on the input data set and the output results of each model (corresponding to L 1- L n ), and then use the gradient of the target detection loss compared to the input data set to calculate the angle cosine loss of the gradient between each sub-model , calculate and sum the losses of each sub-model pairwise to get the integrated loss , and use Iteratively train each sub-model to obtain the final model ; During the testing phase, the test image is input into In the above example, we get multiple outputs (corresponding to O 1- O n ), and input multiple outputs into the arbiter for arbitration to obtain the final output .

[0094] The target detection device based on the integrated model provided by the present invention is described below. The target detection device based on the integrated model described below and the target detection method based on the integrated model described above can be referenced to each other.

[0095] Figure 4 is a schematic diagram of the structure of the target detection device based on the integrated model provided by the present invention, such as Figure 4 As shown, the target detection device based on the integrated model includes a detection module 410 and a decision module 420.

[0096] The integrated detection module 410 is used to input the image to be detected into the differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by training multiple sub-models in parallel with the integrated loss as the training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value between the loss functions of the multiple sub-models; the target detection loss includes at least one of the positioning loss and the classification loss; The decision module 420 is used to make a decision on the multiple detection results to obtain a decision result.

[0097] The target detection device based on the integrated model provided by the embodiment of the present invention calculates the cosine loss of the gradient angle between each sub-model through the target detection loss between the input and output of each sub-model, and weights the cosine loss and the target detection loss of each sub-model to obtain the integrated loss. Finally, differentiated training of multiple sub-models is realized according to the integrated loss. When facing adversarial samples generated based on gradient attacks, the sub-models that are not attacked can well ignore the influence of the adversarial disturbance, weaken the mobility of the adversarial samples in the differentiated sub-models, and thus improve the accuracy of the model output results.

[0098] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the target detection method based on the integrated model, the method comprising: inputting the image to be identified into the differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by training multiple sub-models in parallel with the integrated loss as the training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value of the gradient angle between the loss functions of the multiple sub-models; the target detection loss includes at least one of the positioning loss and the classification loss; and the multiple detection results are adjudicated to obtain the adjudication result.

[0099] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target detection method based on the integrated model provided by the above-mentioned methods, and the method includes: inputting the image to be detected into a differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by parallel training multiple sub-models with integrated loss as the training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value of the gradient angle between the loss functions of multiple sub-models; the target detection loss includes at least one of the positioning loss and the classification loss; and multiple detection results are adjudicated to obtain a judgment result.

[0101] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the target detection method based on an integrated model provided by the above-mentioned methods, the method comprising: inputting an image to be detected into a differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by training multiple sub-models in parallel with an integrated loss as a training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value of the gradient angle between the loss functions of the multiple sub-models; the target detection loss includes at least one of a positioning loss and a classification loss; and multiple detection results are adjudicated to obtain a adjudication result.

[0102] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target detection method based on an integrated model, characterized in that: include: The image to be detected is input into a differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by training multiple sub-models in parallel with an integrated loss as a training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value of the gradient angle between the loss functions of the multiple sub-models; the target detection loss includes at least one of a positioning loss and a classification loss; A decision is made on the multiple detection results to obtain a decision result.

2. The target detection method based on the integrated model according to claim 1, characterized in that: The detection loss is determined based on the positioning loss and the classification loss; The integrated loss is obtained by the following steps: Training the multiple sub-models in parallel, and for each sub-model, calculating the detection loss of the sub-model according to the input and output of the sub-model; Obtaining a cosine loss of a gradient angle between the multiple sub-models according to a gradient of the target detection loss with respect to the input; The cosine loss and the target detection loss of each sub-model are weightedly calculated to obtain the integrated loss.

3. The target detection method based on integrated model according to claim 2, characterized in that: The step of obtaining the cosine loss of the gradient angle between the multiple sub-models according to the gradient of the target detection loss with respect to the input includes: ； in, is the cosine loss, For the i Object detection loss of sub-model About input x The gradient of For the j Object detection loss of sub-model About input x The gradient of N is the number of sub-models.

4. The target detection method based on integrated model according to any one of claims 1 and 2, characterized in that: The differentiated integrated model is obtained by applying an integrated loss function and performing back-propagation training on the multiple sub-models.

5. The target detection method based on integrated model according to any one of claims 1 to 3, characterized in that: The multiple sub-models adopt exactly the same architecture, and the inputs of the multiple sub-models are the same.

6. The target detection method based on integrated model according to claim 1, characterized in that: The adjudicating the multiple detection results to obtain the adjudication result comprises: When a first detection box in the multiple detection results is detected by a first number of sub-models, the first detection box is deleted from the multiple detection results; when a second detection box in the multiple detection results is detected by a second number of sub-models, the second detection boxes detected by the second number of sub-models are merged to obtain a plurality of filtered detection results; wherein the first number does not exceed half of the number of sub-models, and the second number is greater than the first number; The multiple screened test results are combined to obtain the decision result.

7. The target detection method based on integrated model according to claim 6, characterized in that: Different detection results correspond to different two-dimensional matrices, the first detection frame belongs to a subvector of the first two-dimensional matrix, and the second detection frame belongs to a subvector of the second two-dimensional matrix; Deleting the first detection frame from the multiple detection results includes: The first detection frame in the first two-dimensional matrix is ​​reset to zero.

8. The target detection method based on integrated model according to claim 6, characterized in that: The step of fusing the second detection frames detected by the second number of sub-models to obtain a plurality of filtered detection results includes: Get screening test results by any of the following: If there are overlapping detection frames in the multiple detection results, any two overlapping detection frames are recorded as B 1 and B 2 ; Use the following formula to B 1 and B 2 Perform fusion to obtain the filtered test results: ； in, The test results after screening are for B 1 The area of for B 2 area; Alternatively, different detection results correspond to different two-dimensional matrices, the first detection box belongs to a sub-vector of the first two-dimensional matrix, and the second detection box belongs to a sub-vector of the second two-dimensional matrix; any one of a second number of second detection boxes is retained from the multiple detection results, and the other second detection boxes other than the any one second detection box are reset to zero to obtain the multiple filtered detection results.

9. A target detection device based on an integrated model, characterized in that: include: An integrated detection module, used for inputting an image to be detected into a differentiated integrated model for detection to obtain multiple detection results; wherein the differentiated integrated model is obtained by training multiple sub-models in parallel using an integrated loss as a training loss function; the integrated loss is determined based on the target detection loss corresponding to each sub-model and the cosine value of the gradient angle between the loss functions of the multiple sub-models; the target detection loss includes at least one of a positioning loss and a classification loss; The decision module is used to make a decision on the multiple detection results to obtain a decision result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the target detection method based on the integrated model as described in any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the integrated model-based target detection method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the integrated model-based target detection method according to any one of claims 1 to 8 is implemented.

Images