Training methods for object detection networks, object detection methods and devices

By adding a feature point detection loss function and color enhancement techniques during the training of the object detection network, combined with the SSD structure, the problem of poor stability of the object detection network is solved, and the detection accuracy in complex environments and at long distances is improved.

CN115335872BActive Publication Date: 2026-11-17BOE TECHNOLOGY GROUP CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202180000352.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-26
Publication Date
2026-11-17
Estimated Expiration
2041-02-26

AI Technical Summary

Technical Problem

Existing target detection networks have poor stability, especially in complex application scenarios where they are easily affected by interfering objects, making it difficult to accurately detect target objects.

Method used

When training the target detection network, a loss function for the feature point detection location of the target object is added, and the network stability is improved by color enhancement and image cropping techniques. The target detection network is trained in combination with the SSD structure.

Benefits of technology

It improves the accuracy and stability of target detection, especially in complex environments and long-distance detection scenarios, enabling better detection of target objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115335872B_ABST
    Figure CN115335872B_ABST
Patent Text Reader

Abstract

This disclosure provides a training method, object detection method, and apparatus for an object detection network. The training method includes: inputting a training image into the object detection network to be trained to obtain object detection information, including the object detection classification, the detection location of the object's bounding box, and the detection location of the object's feature points; calculating the total loss function of the object detection network to be trained, the total loss function being calculated based on the loss function of the object detection classification, the loss function of the object's bounding box detection location, and the loss function of the object's feature point detection location; and adjusting the parameters of the object detection network to be trained based on the total loss function. In this disclosure, the loss of the target object's feature point detection location is additionally considered when training the object detection network, which helps to improve the quality of detected target objects and reduce the impact of interfering objects on the detection results in complex application scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a training method for an object detection network, an object detection method, and an apparatus. Background Technology

[0002] With the development of computer technology, research on target detection and real-time tracking using computer image processing technology has become increasingly popular. Due to the complexity and variability of application scenarios, high demands are placed on the stability of target detection networks. Summary of the Invention

[0003] This disclosure provides a training method, object detection method, and apparatus for an object detection network, which addresses the problem of poor stability in current object detection networks.

[0004] To solve the above-mentioned technical problems, this disclosure is implemented as follows:

[0005] In a first aspect, embodiments of this disclosure provide a method for training an object detection network, comprising:

[0006] The training image is input into the target detection network to be trained to obtain the detection information of the target object in the training image. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature point of the target object within the detection box.

[0007] The total loss function of the target detection network to be trained is calculated based on the first loss function of the target object detection classification, the second loss function of the detection position of the target object's detection box, and the third loss function of the detection position of the feature point of the target object within the detection box.

[0008] Based on the total loss function of the target detection network to be trained, the parameters of the target detection network to be trained are adjusted to obtain the trained target detection network.

[0009] Optionally, inputting the training image into the target detection network to be trained to obtain the detection information of the target object in the training image includes:

[0010] The training images are input into the first convolutional network of the target detection network to be trained to obtain feature maps at multiple scales;

[0011] The feature maps at each scale are respectively input into the second convolutional network of the target detection network to be trained, so as to obtain the detection information of each pixel position in each feature map;

[0012] The target detection network to be trained includes the first convolutional network and the second convolutional network.

[0013] Optionally, the second convolutional network includes a first convolutional layer with a kernel size of 1*1*n, wherein the first convolutional layer is used to convert the feature map into a first target feature map containing the detection positions of feature points of the target object within the detection box, and n is any positive integer.

[0014] Optionally, the total loss function is calculated using the following formula:

[0015] L = L cls +α1L box +α2L ldm

[0016] Where L is the total loss function, L cls Let L be the first loss function. box Let L be the second loss function. ldm Let α1 and α2 be the weights of the third loss function.

[0017] Optionally, the first loss function is calculated using the following formula:

[0018]

[0019] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, p represents the p-th class, and x ij This indicates whether the i-th candidate box matches the j-th ground truth box. This represents the probability of classifying the i-th candidate box as the p-th class. Pos represents the probability of classifying the i-th candidate box as the background class, Pos represents the set of candidate boxes that match the j-th ground truth box, and Neg represents the set of candidate boxes that do not match the j-th ground truth box.

[0020] Optionally, the second loss function is calculated using the following formula:

[0021]

[0022] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, and x ij This indicates whether the i-th candidate box matches the j-th ground truth box, where Pos represents the set of candidate boxes that match the j-th ground truth box, and smoothness. L1 Denotes the loss function, l bi G represents the detection position of the detection box corresponding to the i-th candidate box. bj Let be the position of the j-th real bounding box.

[0023] Optionally, the third loss function is calculated using the following formula:

[0024]

[0025] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, and x ij This indicates whether the i-th candidate box matches the j-th ground truth box, where Pos represents the set of candidate boxes that match the j-th ground truth box, and smoothness. L1 Denotes the loss function, l li g represents the detection position of the feature point of the target object in the detection box corresponding to the i-th candidate box. lj This represents the true location of the feature points of the target object within the j-th real bounding box.

[0026] Optionally, the detection position of the detection box includes: the offset of the horizontal coordinate of the center point of the detection box relative to the horizontal coordinate of the center point of the candidate box, the offset of the vertical coordinate of the center point of the detection box relative to the vertical coordinate of the center point of the candidate box, the offset of the length of the detection box relative to the length of the candidate box, and the offset of the width of the detection box relative to the width of the candidate box.

[0027] The detection position of the feature point of the target object within the detection box includes: the predicted horizontal coordinate and the predicted vertical coordinate of the feature point of the target object.

[0028] Optionally, before inputting the training image into the target detection network to obtain the detection information of the target object in the training image, the method further includes:

[0029] For each training image to be enhanced, a color mask of the same size as the training image to be enhanced is randomly generated, and the color mask contains only one color.

[0030] The training image to be enhanced is fused with the color mask to obtain a color-enhanced training image, which is then used as the training image input to the target detection network to be trained.

[0031] Optionally, the training image to be enhanced can be fused with the color mask using the following formula:

[0032] img aug =α*color mask +(1-α)*img

[0033] Among them, color mask The generated color mask is represented by α, the training image to be enhanced is represented by α, and the weights are represented by α. aug This represents the training image after color enhancement, where 0 < α < 0.5.

[0034] Optionally, before inputting the training image into the target detection network to obtain the detection information of the target object in the training image, the method further includes:

[0035] For each training image to be enhanced, the training image to be enhanced is converted from the RGB color space to the HSV color space;

[0036] The H channel of the training image to be enhanced, converted to the HSV color space, is randomly transformed to obtain the transformed training image to be enhanced.

[0037] The transformed training image to be enhanced is converted back to the RGB color space to obtain a color-enhanced training image, which is then used as the training image input to the target detection network to be trained.

[0038] Optionally, the target object is a hand, wherein the feature point is a point representing a joint of the target object.

[0039] Secondly, embodiments of this disclosure provide a target detection method, including:

[0040] The image to be detected is input into the target detection network, which outputs the detection location and classification of the detection box of the target object in the image to be detected;

[0041] The target detection network is trained using the target detection network training method described in the first aspect above.

[0042] Optionally, the image to be detected is input into the target detection network, and the detection location and classification of the detection box of the target object in the image to be detected are output, including:

[0043] The image to be detected is input into the target detection network to obtain the detection information of the target object in the image to be detected. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box.

[0044] Based on the detection positions of feature points of the target object within the detection box, the detection position and detection classification of the target object in the image to be detected are output.

[0045] Optionally, before inputting the image to be detected into the target detection network, the method further includes: acquiring the current image to be detected;

[0046] The image to be detected is input into the target detection network, and the detection locations of the target objects in the image to be detected are obtained, including:

[0047] If the target detection network detects the target object in the previous frame of the input image, it records the detection position of the detection box of the target object;

[0048] Based on the detection position of the detection box of the target object in the previous frame of the input image, the position of the cropping box in the current image to be detected is determined, and the detection box is located within the cropping box;

[0049] The current image to be detected is cropped based on the position of the cropping box to obtain the image to be detected to be input into the target detection network.

[0050] Optionally, the center point coordinates of the cutting frame are the same as the center point coordinates of the detection frame, the length of the cutting frame is n times the long side of the detection frame, and the width of the cutting frame is m times the long side of the detection frame.

[0051] Optionally, the aspect ratio of the image to be detected is n:m.

[0052] Optionally, the method further includes:

[0053] If the target detection network does not detect the target object in the previous frame of the input image, the current image to be detected is input into the target detection network as the image to be detected.

[0054] Thirdly, embodiments of this disclosure provide a training apparatus for an object detection network, comprising:

[0055] The prediction module is used to input the training image into the target detection network to be trained to obtain the detection information of the target object in the training image. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box.

[0056] The calculation module is used to calculate the total loss function of the target detection network to be trained. The total loss function is calculated based on the first loss function of the target object detection classification, the second loss function of the detection position of the target object's detection box, and the third loss function of the detection position of the feature point of the target object within the detection box.

[0057] The adjustment module is used to adjust the parameters of the target detection network to be trained according to the total loss function of the target detection network to be trained, so as to obtain the trained target detection network.

[0058] Fourthly, embodiments of this disclosure provide a target detection device, comprising:

[0059] The prediction module is used to input the image to be detected into the target detection network and output the detection location and detection classification of the target object in the image to be detected;

[0060] The target detection network is trained using the target detection network training method described in the first aspect above.

[0061] Fifthly, embodiments of this disclosure provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the training method for the target detection network described in the first aspect, or when the program or instructions are executed by the processor, they implement the steps of the target detection method described in the second aspect.

[0062] In a sixth aspect, embodiments of this disclosure provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the training method for the target detection network of the first aspect described above, or implement the steps of the target detection method of the second aspect described above.

[0063] In this embodiment of the disclosure, when training the target detection network, in addition to considering the detection classification loss and the detection position loss of the detection box of the target object in the training image, the detection position loss of the feature point of the target object is also added. This helps to improve the quality of the detected target object, reduce the influence of interfering objects on the detection results in complex application scenarios, and improve the stability of the target detection network. Attached Figure Description

[0064] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0065] Figure 1 This is a schematic flowchart illustrating a training method for an object detection network according to an embodiment of the present disclosure;

[0066] Figure 2 This is a schematic diagram of the target detection network according to an embodiment of the present disclosure;

[0067] Figure 3 This is a schematic flowchart illustrating a training method for an object detection network according to another embodiment of the present disclosure;

[0068] Figure 4 This is a schematic flowchart illustrating a training method for an object detection network according to yet another embodiment of this disclosure;

[0069] Figure 5 This is a schematic flowchart of a target detection method according to an embodiment of the present disclosure;

[0070] Figure 6 This is a schematic flowchart of a target detection method according to another embodiment of the present disclosure;

[0071] Figure 7 This is a schematic diagram of the structure of a training device for an object detection network according to an embodiment of the present disclosure;

[0072] Figure 8 This is a schematic diagram of the structure of a target detection device according to an embodiment of the present disclosure;

[0073] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0074] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0075] Please refer to Figure 1 This disclosure provides a method for training an object detection network, comprising:

[0076] Step 11: Input the training image into the target detection network to be trained to obtain the detection information of the target object in the training image. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box.

[0077] In this embodiment of the disclosure, the target detection network is used to detect target objects, such as hands or faces.

[0078] In this embodiment, the object detection network processes the input image as follows: A first convolutional network processes the input image, outputting multi-scale feature maps. For each scale of feature map, a second convolutional network processes the feature map, outputting detection information of the target object at each pixel location in each feature map. The detection information includes: the detection position of the target object's detection box, the detection classification of the target object, and the detection positions of feature points of the target object within the detection box. When processing the feature map using the second convolutional network, multiple candidate boxes are predicted at each pixel location in the feature map. For each candidate box, the classification of the candidate box and the detection position of the detection box are predicted, and there is a one-to-one correspondence between the detection boxes and the candidate boxes.

[0079] The target detection network in this embodiment can be an SSD (Single Shot MultiBox Detector) structured target detection network. Six layers of feature maps can be selected. Alternatively, feature maps with other numerical layers can also be selected.

[0080] In this embodiment of the disclosure, the first convolutional network can be any convolutional neural network. For example, it can be based on VGG16 or Mobilenet (a depth-separable convolutional network), with some convolutional layers and fully connected layers removed and several convolutional layers added, and then used as the first convolutional network for the object detection network.

[0081] In this embodiment of the disclosure, the object detection network can calculate the detection classification of the detection box in the following way: compare the candidate box at each pixel position of the feature map with the ground truth box labeled in the training image to obtain the classification of the candidate box. For example, the intersection-union ratio (IU) of the candidate box and the ground truth box can be calculated. If the IU is greater than a preset threshold, the candidate box is considered to be classified as a target object; if it is less than the preset threshold, the candidate box is considered to be classified as background. The classification of the detection box is the same as the classification of the corresponding candidate box.

[0082] Step 12: Calculate the total loss function of the target detection network to be trained. The total loss function is calculated based on the first loss function of the target object detection classification, the second loss function of the detection position of the target object's detection box, and the third loss function of the detection position of the feature point of the target object within the detection box.

[0083] In this embodiment of the disclosure, for each feature map, the total loss function for each pixel location can be calculated.

[0084] Step 13: Adjust the parameters of the target detection network to be trained according to the total loss function of the target detection network to be trained, so as to obtain the trained target detection network.

[0085] In this embodiment of the disclosure, the parameters of the target detection network to be trained are adjusted by integrating the total loss function of each pixel position in each feature map.

[0086] In this embodiment, when training the target detection network, in addition to considering the detection classification loss and the detection position loss of the detection box, an additional detection position loss of the target object's feature points is added. This helps improve the quality of the detected target objects, reduce the influence of interfering objects on the detection results in complex application scenarios, and improve the stability of the target detection network. For situations where the target object is small or its posture is easily variable, such as when detecting hand gestures in a distant image for gesture control, the area of ​​the hand in the image is very small, making it difficult to accurately detect the hand as a target object. This embodiment adds the detection position information of the target object's feature points during target detection network training, enabling the extraction of more features of the target object. This makes the trained network more effective at detecting target objects, increasing the accuracy of target object detection when using the target detection network.

[0087] In this embodiment of the disclosure, optionally, the detection position of the detection box includes: the offset of the horizontal coordinate of the center point of the detection box relative to the horizontal coordinate of the center point of the candidate box, the offset of the vertical coordinate of the center point of the detection box relative to the vertical coordinate of the center point of the candidate box, the offset of the length of the detection box relative to the length of the candidate box, and the offset of the width of the detection box relative to the width of the candidate box.

[0088] In this embodiment of the disclosure, optionally, the detection position of the feature point of the target object within the detection box includes: the predicted abscissa of the feature point of the target object and the predicted ordinate of the feature point of the target object.

[0089] In this embodiment of the disclosure, taking a target detection network for detecting a hand as an example, the structure of the target detection network can be as follows: Figure 2 As shown.

[0090] In this embodiment of the disclosure, optionally, the second convolutional network includes a first convolutional layer with a kernel size of 1*1*n, wherein the first convolutional layer is used to convert the feature map into a first target feature map containing the detection positions of feature points of the target object within the detection box, and n is any positive integer.

[0091] In this embodiment of the disclosure, for the offset of the detection box, the second convolutional network can convert the feature map into a channel number of N. i ×4, size is W i ×H i Feature map, where N iThis represents the number of candidate boxes generated at each pixel location in the i-th layer feature map. Each candidate box receives the offsets of the center coordinates and width and height values ​​of the corresponding detection box. For predicting the location of feature points, the second convolutional network converts the feature map into N channels. i ×N1×2, with a size of W i ×H i The feature map is generated, and each candidate box will obtain the horizontal and vertical coordinates of N1 feature points of the corresponding detection box. The number of feature points can be set as needed. For example, if the target object is a human hand, the number of feature points can be 6, corresponding to 5 finger joints and 1 palm joint.

[0092] In this embodiment of the disclosure, optionally, the total loss function is calculated using the following formula:

[0093] L = L cls +α1L box +α2L ldm

[0094] Where L is the total loss function, L cls Let L be the first loss function. box Let L be the second loss function. ldm Let α1 and α2 be the weights of the third loss function.

[0095] α1 and α2 can be preset or adjusted during the training of the target detection network.

[0096] In this embodiment of the disclosure, optionally, the first loss function is calculated using the following formula:

[0097]

[0098] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, p represents the p-th class, and x ij This indicates whether the i-th candidate box matches the j-th ground truth box. This represents the probability of classifying the i-th candidate box as the p-th class. Pos represents the probability of classifying the i-th candidate box as the background class, Pos represents the set of candidate boxes that match the j-th ground truth box, and Neg represents the set of candidate boxes that do not match the j-th ground truth box.

[0099] In this embodiment of the disclosure, optionally, the second loss function is calculated using the following formula:

[0100]

[0101] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, and x ijThis indicates whether the i-th candidate box matches the j-th ground truth box, where Pos represents the set of candidate boxes that match the j-th ground truth box, and smoothness. L1 Denotes the loss function, l bi G represents the detection position of the detection box corresponding to the i-th candidate box. bj Let be the position of the j-th real bounding box.

[0102] In this embodiment of the disclosure, optionally, the third loss function is calculated using the following formula:

[0103]

[0104] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, and x ij This indicates whether the i-th candidate box matches the j-th ground truth box, where Pos represents the set of candidate boxes that match the j-th ground truth box, and smoothness. L1 Denotes the loss function, l li g represents the detection position of the feature point of the target object in the detection box corresponding to the i-th candidate box. lj This represents the true location of the feature points of the target object within the j-th real bounding box.

[0105] In some embodiments of this disclosure, optionally, the target object can be a hand, wherein the feature point is a point representing a joint portion of the target object.

[0106] In some embodiments of this disclosure, optionally, the target object may also be a face, wherein the feature points are points representing the facial features of the target object.

[0107] In practical applications of object detection networks, the complex and varied ambient lighting, such as the use of special colored lights in exhibition halls, can cause the target objects in the image to appear in different colors, which greatly complicates the detection task.

[0108] To solve the above problems, please refer to... Figure 3 This disclosure provides a method for training an object detection network, comprising:

[0109] Step 31: For each training image to be enhanced, randomly generate a color mask with the same size as the training image to be enhanced, wherein the color mask contains only one color;

[0110] Step 32: Fuse the training image to be enhanced with the color mask to obtain a color-enhanced training image, which is then used as the training image input to the target detection network to be trained.

[0111] In this embodiment of the disclosure, optionally, the training image to be enhanced is fused with the color mask using the following formula:

[0112] img aug =α*color mask +(1-α)*img

[0113] Among them, color mask The generated color mask is represented by α, the training image to be enhanced is represented by α, and the weights are represented by α. aug This represents the training image after color enhancement, where 0 < α < 0.5.

[0114] Step 33: Input the training image into the target detection network to be trained to obtain the detection information of the target object. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box.

[0115] Step 34: Calculate the total loss function of the target detection network to be trained. The total loss function is calculated based on the first loss function of the target object detection classification, the second loss function of the detection position of the target object's detection box, and the third loss function of the detection position of the feature point of the target object within the detection box.

[0116] Step 35: Adjust the parameters of the target detection network to be trained according to the total loss function of the target detection network to be trained, so as to obtain the trained target detection network.

[0117] In this embodiment of the disclosure, by adjusting the color of the training image, the trained object detection network can be made suitable for different lighting environments, thereby improving the stability of the object detection network.

[0118] In this embodiment, other methods can also be used to adjust the color of the training image; please refer to [reference needed]. Figure 4 This disclosure provides a method for training an object detection network, comprising:

[0119] Step 41: For each training image to be enhanced, convert the training image to be enhanced from the RGB color space to the HSV color space;

[0120] Step 42: Randomly transform the H channel of the training image to be enhanced, which has been converted to the HSV color space, to obtain the transformed training image to be enhanced.

[0121] Optionally, performing random transformation on the H channels of the training image to be enhanced, converted to the HSV color space, includes performing a linear transformation on the H channels of the training image to be enhanced, converted to the HSV color space.

[0122] Step 43: Convert the transformed training image to be enhanced back to the RGB color space to obtain the color-enhanced training image, which is then used as the training image input to the target detection network to be trained.

[0123] Step 44: Input the training image into the target detection network to be trained to obtain the detection information of the target object. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box.

[0124] Step 45: Calculate the total loss function of the target detection network to be trained. The total loss function is calculated based on the first loss function of the target object detection classification, the second loss function of the detection position of the target object's detection box, and the third loss function of the detection position of the feature point of the target object within the detection box.

[0125] Step 46: Adjust the parameters of the target detection network to be trained according to the total loss function of the target detection network to be trained, so as to obtain the trained target detection network.

[0126] Please refer to Figure 5 This disclosure also provides a target detection method, including:

[0127] Step 51: Input the image to be detected into the target detection network, and output the detection position and detection classification of the target object in the image to be detected; wherein, the target detection network is trained using the target detection network training method described in any of the above embodiments.

[0128] In some embodiments of this disclosure, when training an object detection network, it is necessary to output three types of data: the detection classification of the object's detection box, the position of the detection box, and the detection position of the feature points of the object within the detection box, in order to optimize the network parameters. However, when actually using the object detection network, only the position and detection classification of the detection box need to be output, that is, the detection position of the feature points of the object within the detection box is not used.

[0129] In other embodiments, the detection location of feature points of the target object within the detection box can also be used. That is, the image to be detected is input into the target detection network, and the detection location and detection classification of the target object in the image to be detected are output, including:

[0130] The image to be detected is input into the target detection network to obtain the detection information of the target object in the image to be detected. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box.

[0131] Based on the detected positions of feature points of the target object within the detection box, the detection position and classification of the detection box of the target object in the image to be detected are output. For example, if no feature points are detected within the detection box, or if key points are detected but the analysis results indicate that they are not key points of the target object, then it can be determined that the detection box does not belong to the target object.

[0132] In long-distance target detection scenarios, the difficulty of detecting some target objects increases significantly. For example, in a scenario where a hand simulates a mouse, it's necessary to detect a clenched fist gesture to trigger a "click" action. However, when performing long-distance detection, the area of ​​a clenched fist is significantly smaller than that of an open hand, making the detection task much more challenging. To address these issues, please refer to... Figure 6 This disclosure also provides a target detection method, including:

[0133] Step 61: For each image to be input into the target detection network, check whether the target detection network detected the target object in the previous frame of the input image; if yes, proceed to step 62; otherwise, proceed to step 65.

[0134] Step 62: If the target detection network detects the target object in the previous frame of the input image, record the detection position of the detection box of the target object;

[0135] Step 63: Determine the position of the cropping box in the current image to be detected based on the detection position of the detection box of the target object in the previous frame input image, wherein the detection box is located within the cropping box;

[0136] Optionally, the center point coordinates of the cropping frame are the same as the center point coordinates of the detection frame. The length of the cropping frame is n times the longer side of the detection frame, and the width of the cropping frame is m times the longer side of the detection frame. Both m and n are positive integers. For example, assuming the longer side of the detection frame is x, the size of the cropping frame is 4x × 3x.

[0137] Optionally, the aspect ratio of the image to be detected is n:m, such as 4:3 or 16:9.

[0138] Step 64: Crop the current image to be detected based on the position of the cropping box to obtain the image to be detected to be input into the target detection network.

[0139] Step 65: If the target detection network does not detect the target object in the previous frame of the input image, then the current frame image is input to the target detection network as the image to be detected.

[0140] In this embodiment of the present disclosure, after a target object is detected in the captured image of the current frame, the area near the detection box is cropped out as the input of the target detection network in the detection of the next frame. This can increase the area ratio of the target object in the entire image at a distance, thereby effectively improving the detection accuracy at a distance and improving the stability of the target detection network in each frame detection process.

[0141] The target detection network in this embodiment can be an SSD (Single Shot MultiBox Detector) structured target detection network.

[0142] In this embodiment of the disclosure, if the target object is a hand, the user can first extend their palm to "activate" the target detection network during the interaction process; after the target detection network detects the user's palm, it will stabilize and perform detection in the vicinity of that area; after the "activation" operation is completed, the user can interact with the computer through various other gestures; when the user finds that the interaction between them and the computer is interrupted, they can "activate" the algorithm again by operating their palm.

[0143] Please refer to Figure 7 This disclosure provides a target detection device 70, comprising:

[0144] The prediction module 71 is used to input the training image into the target detection network to be trained to obtain the detection information of the target object in the training image. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box.

[0145] The calculation module 72 is used to calculate the total loss function of the target detection network to be trained. The total loss function is calculated based on the first loss function of the target object detection classification, the second loss function of the detection position of the target object's detection box, and the third loss function of the detection position of the feature point of the target object within the detection box.

[0146] The adjustment module 73 is used to adjust the parameters of the target detection network to be trained according to the total loss function of the target detection network to be trained, so as to obtain the trained target detection network.

[0147] Optionally, the detection position of the detection box includes: the offset of the horizontal coordinate of the center point of the detection box relative to the horizontal coordinate of the center point of the candidate box, the offset of the vertical coordinate of the center point of the detection box relative to the vertical coordinate of the center point of the candidate box, the offset of the length of the detection box relative to the length of the candidate box, and the offset of the width of the detection box relative to the width of the candidate box.

[0148] The detection position of the feature points of the target object within the detection box includes: the predicted horizontal coordinate and the predicted vertical coordinate of the feature points of the target object.

[0149] Optionally, the prediction module is used to input the training image into the first convolutional network of the target detection network to be trained to obtain feature maps at multiple scales; and to input the feature maps at each scale into the second convolutional network of the target detection network to be trained to obtain detection information of the detection box at each pixel position in each feature map, wherein the target detection network to be trained includes the first convolutional network and the second convolutional network.

[0150] Optionally, the second convolutional network includes a first convolutional layer with a kernel size of 1*1*n, wherein the first convolutional layer is used to convert the feature map into a first target feature map containing the detection positions of feature points of the target object within the detection box, and n is any positive integer.

[0151] Optionally, the total loss function is calculated using the following formula:

[0152] L = L cls +α1L box +α2L ldm

[0153] Where L is the total loss function, L cls Let L be the first loss function. box Let L be the second loss function. ldm Let α1 and α2 be the weights of the third loss function.

[0154] Optionally, the first loss function is calculated using the following formula:

[0155]

[0156] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, p represents the p-th class, and x ij This indicates whether the i-th candidate box matches the j-th ground truth box. This represents the probability of classifying the i-th candidate box as the p-th class. Pos represents the probability of classifying the i-th candidate box as the background class, Pos represents the set of candidate boxes that match the j-th ground truth box, and Neg represents the set of candidate boxes that do not match the j-th ground truth box.

[0157] Optionally, the second loss function is calculated using the following formula:

[0158]

[0159] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, and x ij This indicates whether the i-th candidate box matches the j-th ground truth box, where Pos represents the set of candidate boxes that match the j-th ground truth box, and smoothness. L1 Denotes the loss function, l bi G represents the detection position of the detection box corresponding to the i-th candidate box. bj Let be the position of the j-th real bounding box.

[0160] Optionally, the third loss function is calculated using the following formula:

[0161]

[0162] Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, and x ij This indicates whether the i-th candidate box matches the j-th ground truth box, where Pos represents the set of candidate boxes that match the j-th ground truth box, and smoothness. L1 Denotes the loss function, l li g represents the detection position of the feature point of the target object in the detection box corresponding to the i-th candidate box. lj This represents the true location of the feature points of the target object within the j-th real bounding box.

[0163] Optionally, the training device for the object detection network further includes:

[0164] The generation module is used to randomly generate a color mask of the same size as the training image to be enhanced for each training image to be enhanced, wherein the color mask contains only one color.

[0165] The fusion module is used to fuse the training image to be enhanced with the color mask to obtain a color-enhanced training image, which is then used as the training image input to the target detection network to be trained.

[0166] Optionally, the fusion module is used to fuse the training image to be enhanced with the color mask using the following formula:

[0167] img aug =α*color mask +(1-α)*img

[0168] Among them, color mask The generated color mask is represented by α, the training image to be enhanced is represented by α, and the weights are represented by α. aug This represents the training image after color enhancement, where 0 < α < 0.5.

[0169] Optionally, the training device for the object detection network further includes:

[0170] The first conversion module is used to convert each training image to be enhanced from the RGB color space to the HSV color space.

[0171] The transformation module is used to randomly transform the H channel of the training image to be enhanced after it has been converted to the HSV color space, so as to obtain the transformed training image to be enhanced.

[0172] The second conversion module is used to convert the transformed training image to be enhanced back to the RGB color space to obtain a color-enhanced training image, which is then used as the training image input to the target detection network to be trained.

[0173] Optionally, the transformation module is used to perform a linear transformation on the H channel of the training image to be enhanced, which has been converted to the HSV color space.

[0174] Optionally, the target object is a hand, wherein the feature point is a point representing a joint of the target object.

[0175] Please refer to Figure 8 This disclosure provides a target detection device 80, comprising:

[0176] The prediction module 81 is used to input the image to be detected into the target detection network and output the detection position and detection classification of the detection box of the target object in the image to be detected; wherein the target detection network is trained using the training method of the target detection network described in the first aspect above.

[0177] Optionally, the prediction module is configured to: if the target detection network detects the target object in the previous frame of the input image, record the detection position of the detection box of the target object; determine the cropping box in the current image to be detected as u-shaped based on the detection position of the target object in the previous frame of the input image, wherein the detection box is located within the cropping box; and crop the current image to be detected based on the u-shaped cropping box to obtain the image to be input to the target detection network.

[0178] Optionally, the center point coordinates of the cutting frame are the same as the center point coordinates of the detection frame, the length of the cutting frame is n times the long side of the detection frame, and the width of the cutting frame is m times the long side of the detection frame.

[0179] Optionally, the aspect ratio of the image to be detected is n:m.

[0180] Optionally, the prediction module is configured to input the current image to be detected as the image to be detected into the target detection network if the target detection network does not detect the target object in the previous frame of the input image.

[0181] like Figure 9 As shown, this application embodiment also provides an electronic device 90, including a processor 91, a memory 92, and a program or instructions stored in the memory 92 and executable on the processor 91. When the program or instructions are executed by the processor 91, they implement the various processes of the above-described target detection network training method embodiment and achieve the same technical effect. Alternatively, when the program or instructions are executed by the processor 91, they implement the various processes of the above-described target detection method embodiment and achieve the same technical effect.

[0182] This application also provides a readable storage medium storing a program or instructions. When executed by a processor, the program or instructions implement the various processes of the above-described target detection network training method embodiments, or, when executed by a processor, implement the various processes of the above-described target detection method embodiments, achieving the same technical effect. To avoid repetition, further details are omitted here. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0183] The embodiments of this disclosure have been described above with reference to the accompanying drawings. However, this disclosure is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this disclosure without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this disclosure.

Claims

1. A training method for an object detection network, characterized in that, include: The training image is input into the target detection network to be trained to obtain the detection information of the target objects in the training image. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box. The process of inputting the training image into the target detection network to obtain the detection information of the target objects in the training image includes: inputting the training image into a first convolutional network of the target detection network to be trained to obtain feature maps at multiple scales; inputting each scale feature map into a second convolutional network of the target detection network to be trained to obtain the detection information of each pixel position in each feature map; wherein the target detection network to be trained includes the first convolutional network and the second convolutional network. The total loss function of the target detection network to be trained is calculated based on a first loss function for the detection and classification of the target object, a second loss function for the detection position of the target object's detection box, and a third loss function for the detection position of the feature point of the target object within the detection box; wherein, calculating the total loss function of the target detection network to be trained includes: for each feature map, calculating the total loss function for each pixel position; The parameters of the target detection network to be trained are adjusted according to the total loss function of the target detection network to be trained, to obtain the trained target detection network; wherein, adjusting the parameters of the target detection network to be trained according to the total loss function of the target detection network to be trained includes: adjusting the parameters of the target detection network to be trained by combining the total loss function of each pixel position in each feature map; The process of inputting the training image into the target detection network to obtain the detection information of the target object in the training image also includes: For each training image to be enhanced, a color mask of the same size as the training image to be enhanced is randomly generated, and the color mask contains only one color. The training image to be enhanced is fused with the color mask to obtain a color-enhanced training image, which is then used as the training image input to the target detection network to be trained. The following formula is used to fuse the training image to be enhanced with the color mask: ; in, This represents the generated color mask, and `img` represents the training image to be enhanced. Indicates weight, This represents the training image after color enhancement. .

2. The method according to claim 1, characterized in that, The second convolutional network includes a first convolutional layer with a kernel size of 1*1*n, wherein the first convolutional layer is used to convert the feature map into a first target feature map containing the detection positions of feature points of the target object within the detection box, and n is any positive integer.

3. The method according to claim 1, characterized in that, The total loss function is calculated using the following formula: ; in, Let the total loss function be... Let the first loss function be... For the second loss function, For the third loss function, and For weights.

4. The method according to claim 3, characterized in that, The first loss function is calculated using the following formula: ; Where i represents the i-th candidate bounding box, j represents the j-th ground truth bounding box, and p represents the p-th class. This indicates whether the i-th candidate box matches the j-th ground truth box. This represents the probability of classifying the i-th candidate box as the p-th class. Pos represents the probability of classifying the i-th candidate box as the background class, Pos represents the set of candidate boxes that match the j-th ground truth box, and Neg represents the set of candidate boxes that do not match the j-th ground truth box.

5. The method according to claim 3, characterized in that, The second loss function is calculated using the following formula: ; Where i represents the i-th candidate bounding box, and j represents the j-th ground truth bounding box. This indicates whether the i-th candidate box matches the j-th ground truth box, and Pos represents the set of candidate boxes that match the j-th ground truth box. Represents the loss function. This indicates the detection position of the detection box corresponding to the i-th candidate box. Let be the position of the j-th real bounding box.

6. The method according to claim 3, characterized in that, The third loss function is calculated using the following formula: ; Where i represents the i-th candidate bounding box, and j represents the j-th ground truth bounding box. This indicates whether the i-th candidate box matches the j-th ground truth box, and Pos represents the set of candidate boxes that match the j-th ground truth box. Represents the loss function. This represents the detected position of the feature point of the target object in the detection box corresponding to the i-th candidate box. This represents the true location of the feature points of the target object within the j-th real bounding box.

7. The method according to claim 1, characterized in that, The detection position of the detection box includes: the offset of the horizontal coordinate of the center point of the detection box relative to the horizontal coordinate of the center point of the candidate box, the offset of the vertical coordinate of the center point of the detection box relative to the vertical coordinate of the center point of the candidate box, the offset of the length of the detection box relative to the length of the candidate box, and the offset of the width of the detection box relative to the width of the candidate box. The detection position of the feature point of the target object within the detection box includes: the predicted horizontal coordinate and the predicted vertical coordinate of the feature point of the target object.

8. The method according to any one of claims 1-7, characterized in that, Before inputting the training image into the target detection network to obtain the detection information of the target object in the training image, the process also includes: For each training image to be enhanced, the training image to be enhanced is converted from the RGB color space to the HSV color space; The H channel of the training image to be enhanced, converted to the HSV color space, is randomly transformed to obtain the transformed training image to be enhanced. The transformed training image to be enhanced is converted back to the RGB color space to obtain a color-enhanced training image, which is then used as the training image input to the target detection network to be trained.

9. The method according to any one of claims 1-7, characterized in that, The target object is a hand, and the feature point is a point representing the joint of the target object.

10. A target detection method, characterized in that, include: The image to be detected is input into the target detection network, which outputs the detection location and classification of the detection box of the target object in the image to be detected; The target detection network is trained using the target detection network training method described in any one of claims 1-9.

11. The method according to claim 10, characterized in that, The image to be detected is input into the target detection network, and the network outputs the detection location and classification of the target object in the image, including: The image to be detected is input into the target detection network to obtain the detection information of the target object in the image to be detected. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box. Based on the detection positions of feature points of the target object within the detection box, the detection position and detection classification of the target object in the image to be detected are output.

12. The method according to claim 10, characterized in that, Before inputting the image to be detected into the target detection network, the process also includes: acquiring the current image to be detected; The image to be detected is input into the target detection network, and the detection locations of the target objects in the image to be detected are obtained, including: If the target detection network detects the target object in the previous frame of the input image, it records the detection position of the detection box of the target object; Based on the detection position of the detection box of the target object in the previous frame of the input image, the position of the cropping box in the current image to be detected is determined, and the detection box is located within the cropping box; The current image to be detected is cropped based on the position of the cropping box to obtain the image to be detected to be input into the target detection network.

13. The method according to claim 12, characterized in that, The center point coordinates of the cutting frame are the same as the center point coordinates of the detection frame. The length of the cutting frame is n times the long side of the detection frame, and the width of the cutting frame is m times the long side of the detection frame.

14. The method according to claim 13, characterized in that, The aspect ratio of the image to be detected is n:m.

15. The method according to claim 12, characterized in that, Also includes: If the target detection network does not detect the target object in the previous frame of the input image, the current image to be detected is input into the target detection network as the image to be detected.

16. A training device for an object detection network, characterized in that, include: A prediction module is used to input a training image into a target detection network to be trained to obtain detection information of target objects in the training image. The detection information includes the detection classification of the target object, the detection position of the detection box of the target object, and the detection position of the feature points of the target object within the detection box. The process of inputting the training image into the target detection network to obtain the detection information of target objects in the training image includes: inputting the training image into a first convolutional network of the target detection network to be trained to obtain feature maps at multiple scales; and inputting each scale feature map into a second convolutional network of the target detection network to obtain detection information for the position of each pixel in each feature map. The target detection network to be trained includes the first convolutional network and the second convolutional network. The calculation module is used to calculate the total loss function of the target detection network to be trained. The total loss function is calculated based on the first loss function of the target object detection classification, the second loss function of the detection position of the target object's detection box, and the third loss function of the detection position of the feature point of the target object within the detection box. The calculation of the total loss function of the target detection network to be trained includes: calculating the total loss function for each pixel position for each feature map. An adjustment module is used to adjust the parameters of the target detection network to be trained according to the total loss function of the target detection network to be trained, so as to obtain the trained target detection network; wherein, adjusting the parameters of the target detection network to be trained according to the total loss function of the target detection network to be trained includes: adjusting the parameters of the target detection network to be trained by combining the total loss function of each pixel position in each feature map; The generation module is used to randomly generate a color mask of the same size as the training image to be enhanced for each training image to be enhanced, wherein the color mask contains only one color. The fusion module is used to fuse the training image to be enhanced with the color mask to obtain a color-enhanced training image, which is then used as the training image input to the target detection network to be trained. The training image to be enhanced is fused with the color mask using the following formula: ; in, This represents the generated color mask, and `img` represents the training image to be enhanced. Indicates weight, This represents the training image after color enhancement. .

17. A target detection device, characterized in that, include: The prediction module is used to input the image to be detected into the target detection network and output the detection location and detection classification of the target object in the image to be detected; The target detection network is trained using the target detection network training method described in any one of claims 1-9.

18. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the training method for the object detection network as claimed in any one of claims 1 to 9, or, when the program or instructions are executed by the processor, they implement the steps of the object detection method as claimed in any one of claims 10 to 15.

19. A non-transient computer-readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the training method for the target detection network as described in any one of claims 1 to 9, or implement the steps of the target detection method as described in any one of claims 10 to 15.

Citation Information

Patent Citations

  • Natural image matting method based on deep learning

    CN111161277A

  • Target detection method and device, electronic equipment and computer readable storage medium

    CN111199230A

  • Face detection and alignment method and device

    CN111738077A