Target detection method, electronic device, and storage medium

Abstract

The application discloses a target detection method, an electronic device and a storage medium, and relates to the technical field of target detection. The target detection method comprises the following steps: acquiring a to-be-detected image comprising a target object; performing target detection on the to-be-detected image by using a target detection model to obtain a detection frame of the target object and a plurality of key points corresponding to the detection frame; and the target detection model is obtained by training based on the positions of training detection frames of training images and limiting the positions of a plurality of training key points corresponding to the training detection frames. In the manner, the target detection accuracy of the target detection model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a target detection method, an electronic device and a storage medium. BACKGROUND

[0002] With the continuous development of society and the continuous improvement of technology, people's requirements for technology have been beyond the detection of the category and position of the object in the image, and there is a higher requirement for the posture of the object in the picture or video and accurate positioning.

[0003] Target detection is often performed by key point detection, and key point detection is to find the coordinates of these points from the image by computer. As a basic task in the field of computer vision, key point detection is of great significance to high-level tasks such as recognition and classification.

[0004] Most of the key point detection algorithms in the existing target detection are based on single target, that is, the key frame of a single target is given to cut out the target from the original image for key point detection, which is low in efficiency and leads to low target detection accuracy. SUMMARY

[0005] The present application provides a target detection method, an electronic device and a storage medium to solve the problem of low target detection accuracy.

[0006] To solve the above technical problems, the present application provides a target detection method, comprising: obtaining a to-be-detected image comprising a target object; performing target detection on the to-be-detected image by a target detection model to obtain a detection frame of the target object and a plurality of key points corresponding to the detection frame; wherein the target detection model is trained based on the positions of the training detection frames of the training images to limit the positions of the corresponding plurality of training key points.

[0007] The target detection model comprises a feature extraction subnetwork, an adjustment subnetwork and a detection subnetwork which are connected to each other; the target detection model is used to perform target detection on the to-be-detected image to obtain the detection frame of the target object and the plurality of key points corresponding to the detection frame, which comprises: sequentially passing the to-be-detected image through the feature extraction subnetwork, the adjustment subnetwork and the detection subnetwork to perform target detection on the to-be-detected image to obtain a plurality of initial detection frames of the target object and initial key points corresponding to each initial detection frame; and performing deduplication processing on the plurality of initial detection frames and the plurality of initial key points to obtain the detection frame of the target object and the plurality of key points corresponding to the detection frame.

[0008] The detection sub-network includes a plurality of detection heads, and each detection head is connected with the adjustment sub-network; target detection is performed on the to-be-detected image through the feature extraction sub-network, the adjustment sub-network and the detection sub-network in sequence to obtain a plurality of initial detection boxes of the target object and initial key points corresponding to each initial detection box, including: performing feature extraction on the to-be-detected image through the feature extraction sub-network to obtain a feature map of the to-be-detected image; inputting the feature map into the adjustment sub-network for feature adjustment to obtain a plurality of features of different scales; inputting the plurality of features of different scales into corresponding detection heads for detection processing of different multiples to obtain the plurality of initial detection boxes and the initial key points corresponding to each initial detection box.

[0009] The to-be-detected image including the target object is obtained, and the to-be-detected image is obtained before, including: obtaining a training image; performing target detection on the training image through a target detection model in training to obtain a plurality of training detection boxes and a plurality of training key points corresponding to each training detection box; limiting the positions of the training key points based on the training detection boxes; training the target detection model in training by using the limited training key points to obtain a trained target detection model.

[0010] The positions of the training key points are limited based on the positions of the training detection boxes, including: limiting the positions of the training key points within a preset range of the training detection boxes through a compression function.

[0011] The positions of the training key points are limited within the preset range of the training detection boxes through the compression function, and the preset range of the training detection box is determined based on a center point of the training detection box and a sum of a width and a height of the training detection box with a first preset proportion.

[0012] The target detection model in training is trained by using the limited training key points to obtain the trained target detection model, including: obtaining a plurality of standard key points; determining an expanded width-height range based on the product of the width and the height of the training detection box with a second preset proportion; expanding the limited training key points based on the expanded width-height range to obtain a plurality of training key boxes, with each limited training key point as a center; expanding the standard key points based on the expanded width-height range to obtain a plurality of standard key boxes, with each standard key point as a center; training the target detection model in training by using the difference between each training key box and the corresponding standard key box to obtain the trained target detection model.

[0013] The target detection model in training is trained by using the difference between each training key box and the corresponding standard key box to obtain the trained target detection model, including: training the target detection model in training by using the angle difference, the distance difference, the shape difference and the intersection over union between the training key box and the standard key box through a loss function to obtain the trained target detection model.

[0014] Among them, the object detection method is applied to image recognition; the object detection model is used to detect objects in the image to be detected, and the detection box of the object and the multiple key points corresponding to the detection box are obtained. Then, the object is identified based on the detection box of the object and the multiple key points corresponding to the detection box to obtain the type of the object.

[0015] To address the aforementioned technical problems, the present invention also provides an electronic device comprising: a memory and a processor coupled to each other, wherein the processor is configured to execute program instructions stored in the memory to implement any of the target detection methods described above.

[0016] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing program data that can be executed to implement the target detection method as described above.

[0017] The beneficial effects of this invention are as follows: Unlike existing technologies, this invention trains a target detection model by restricting the positions of multiple training keypoints corresponding to the training detection boxes based on the positions of the training images. This constrains the prediction of keypoint positions, making the learned positions of keypoints related to the detection boxes. This accelerates the learning efficiency of keypoint positions and makes the learned positions more reasonable, thereby improving the target detection accuracy of the target detection model. Furthermore, by using the target detection model to perform target detection on the image to be detected, obtaining the target object's detection box and multiple keypoints corresponding to the detection box, the information richness of the target detection results can be improved, making it applicable to various target detection scenarios and enhancing the applicability of the target detection method. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an embodiment of the target detection method provided by the present invention;

[0019] Figure 2 This is a flowchart illustrating another embodiment of the target detection method provided by the present invention;

[0020] Figure 3 This is a schematic diagram of the structure of one implementation of the target detection model;

[0021] Figure 4 This is a schematic diagram of another implementation of the target detection model;

[0022] Figure 5 This is a schematic diagram of one implementation method for the target detection result of the image to be detected;

[0023] Figure 6is a schematic diagram of another embodiment of a target detection result of a to-be-detected image;

[0024] Figure 7 is a schematic diagram of a training process of an embodiment of a target detection model of the present application;

[0025] Figure 8 is a schematic diagram of an embodiment of training a key frame and a corresponding standard key frame;

[0026] Figure 9 is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0027] Figure 10 is a structural schematic diagram of an embodiment of a computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0029] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of a target detection method provided by the present application.

[0030] Step S11: obtaining a to-be-detected image including a target object.

[0031] The target object of the present embodiment can include a vehicle, a face, a human body, an animal, a tool, a house, a flag, etc. Any object is not limited here.

[0032] In the present step, a to-be-detected image including a target object is obtained. The number of target objects in the to-be-detected image can be one or more.

[0033] In a specific application scenario, the present embodiment can be applied to target detection of a vehicle photographed on a road, so as to perform vehicle recognition. In a specific application scenario, the present embodiment can be applied to target detection of a pedestrian photographed on a road, so as to perform identity recognition. In a specific application scenario, the present embodiment can be applied to target detection of an animal photographed, so as to perform classification, etc. The actual situation can be determined, which is not limited here.

[0034] Step S12: performing target detection on the to-be-detected image by a target detection model to obtain a detection frame of the target object and a plurality of key points corresponding to the detection frame.

[0035] The target detection model is used to perform target detection on the to-be-detected image, to obtain a detection frame of the target object and a plurality of key points corresponding to the detection frame.

[0036] The detection frame is used to mark the target object as a whole on the to-be-detected image. The key points refer to points on the target object that have key features. In a specific application scenario, when the target object is a vehicle, the key points can include a vehicle light key point, a license plate key point, a wheel key point, or a roof key point. In another specific application scenario, when the target object is a face, the key points can include a nose key point, a left eye key point, a right eye key point, or a lip key point.

[0037] The embodiment simultaneously predicts the detection frame of the target object and the plurality of key points corresponding to the detection frame, which can improve the information richness of the target detection result, and is suitable for a variety of target detection scenarios, thereby improving the applicability of the target detection method.

[0038] The target detection model is trained based on the positions of the training detection frames and the positions of the plurality of training key points corresponding to the training detection frames. The training detection frames and the plurality of training key points corresponding to the training detection frames are obtained by performing target detection on the training images by the target detection model in the training. The target detection method is the same as the target detection method of the to-be-detected image in the embodiment.

[0039] Since the detection frame is used to mark the target object as a whole, and the key points are points on the target object that have key features, if the positions of the key points are not limited, the learned coordinates of the key points will be far beyond the range of the detection frame, resulting in unreasonable corresponding relationship and reducing the accuracy of target detection. The embodiment adjusts the positions of the plurality of training key points corresponding to the training detection frames based on the positions of the training detection frames, thereby constraining the prediction of the positions of the key points, making the learned positions of the key points related to the detection frame, and accelerating the learning efficiency of the positions of the key points and making the learning of the positions of the key points more reasonable.

[0040] Through the above steps, the object detection method in this embodiment restricts the positions of multiple training keypoints corresponding to the training detection boxes based on the positions of the training detection boxes in the training images, thereby training an object detection model. This constrains the prediction of keypoint positions, making the learned positions of keypoints related to the detection boxes. This accelerates the learning efficiency of keypoint positions and makes the learned positions more reasonable, thus improving the object detection accuracy of the object detection model. Furthermore, by using the object detection model to perform object detection on the image to be detected, obtaining the detection boxes of the target objects and the multiple keypoints corresponding to the detection boxes, the information richness of the object detection results can be improved, making it applicable to various object detection scenarios and enhancing the applicability of the object detection method.

[0041] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the target detection method provided by the present invention.

[0042] Step S21: Obtain the image to be detected, which includes the target object.

[0043] This step is the same as step S11 in the previous embodiment, please refer to the previous text, and will not be repeated here.

[0044] Step S22: The target detection is performed on the image to be detected sequentially through the feature extraction subnetwork, the adjustment subnetwork and the detection subnetwork to obtain multiple initial detection boxes of the target object and the initial key points corresponding to each initial detection box.

[0045] Please see Figure 3 , Figure 3 This is a schematic diagram of one implementation method of the target detection model.

[0046] The target detection model 30 in this embodiment includes a cascaded feature extraction subnetwork 31, an adjustment subnetwork 32, and a detection subnetwork 33.

[0047] The feature extraction subnetwork 31 is used to extract features from the image to be detected and generate a feature map. The feature extraction subnetwork 31 may include a residual network (ResNet), a convolutional neural network (Visual Geometry Group), a DarkNet network (an open-source deep learning framework), or other deep learning feature extraction networks, which are not limited here.

[0048] The adjustment subnetwork 32 is used for feature adjustment, which can reduce or adjust the features to better adapt to task requirements. The adjustment subnetwork 32 can use convolutional layers, pooling layers, or fully connected layers, etc., without limitation.

[0049] The detection sub-network 33 is the last layer of the target detection model 30, which can be a classifier or a regressor. The detection sub-network 33 generates the final output by inputting the features processed by the adjustment sub-network 32, i.e., the initial detection frame and the corresponding initial key points.

[0050] The structure of the detection sub-network 33 can be different according to different tasks. For example, when the target detection is applied to an image classification task, the detection sub-network 33 can use a softmax classifier; when the target detection is applied to a pure target detection task, the detection sub-network 33 can use a bounding box regressor and a classifier, etc.

[0051] In the embodiment, after the image to be detected is input into the target detection model 30, the image to be detected is first subjected to feature extraction by the feature extraction sub-network 31 to obtain a feature map of the image to be detected; then the feature map is input into the adjustment sub-network 32 for feature adjustment to obtain adjusted features; finally, the adjusted features are input into the detection sub-network 33 for feature detection to obtain multiple initial detection frames of the target object and initial key points corresponding to each initial detection frame.

[0052] In other embodiments, the target detection model can further include other structures for feature extraction, adjustment, and detection, which are not limited herein.

[0053] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of another embodiment of the target detection model.

[0054] The target detection model 40 of the embodiment includes a feature extraction sub-network 41, an adjustment sub-network 42, and a detection sub-network 43 which are cascaded with each other. The feature extraction sub-network 41 and the adjustment sub-network 42 of the embodiment are similar to the feature extraction sub-network 31 and the adjustment sub-network 32 of the foregoing embodiment, which can be referred to the foregoing.

[0055] The detection sub-network 43 includes multiple detection heads, each of which is connected with the adjustment sub-network 42. Each detection head is connected with a corresponding key point branch and a detection branch. The detection head is used to down-sample the features adjusted by the adjustment sub-network 42. The key point branch is used to generate initial key points based on the down-sampling result of the corresponding detection head. The detection branch is used to generate an initial detection frame based on the down-sampling result of the corresponding detection head. The specific number of the detection head and the corresponding key point branch and detection branch can be set based on actual needs, for example, 2, 3, 4, 6, or 8, etc. which are not limited herein.

[0056] The embodiment takes 4 detection heads, 4 key point branches and 4 detection branches as an example for illustration. The detection sub-network 43 of the embodiment includes a first detection head 44, a second detection head 45, a third detection head 46 and a fourth detection head 47. The first detection head 44, the second detection head 45, the third detection head 46 and the fourth detection head 47 are respectively connected with the adjustment sub-network 42. That is, the first detection head 44, the second detection head 45, the third detection head 46 and the fourth detection head 47 are similar to the parallel relationship.

[0057] The first detection head 44 is further connected with a first key point branch 441 and a first detection branch 442; the second detection head 45 is further connected with a second key point branch 451 and a second detection branch 452; the third detection head 46 is further connected with a third key point branch 461 and a third detection branch 462; and the fourth detection head 47 is further connected with a fourth key point branch 471 and a fourth detection branch 472.

[0058] The adjustment sub-network 42 of the embodiment can include the structure of a feature pyramid (FPN, Feature Pyramid Networks) + a region proposal network (RPN, Region Proposal Network) for enhancing the aggregation ability of different levels of features. The feature pyramid can construct a series of images or feature maps of different scales. In the embodiment, the feature pyramid is based on the feature map, and four different scales of features are constructed based on the number of detection heads to correspond to the input into the first detection head 44, the second detection head 45, the third detection head 46 and the fourth detection head 47, respectively, for different multiple detection processing, i.e. down-sampling processing.

[0059] In a specific application scenario, the first detection head 44 can be down-sampled by 8 times. The second detection head 45 can be down-sampled by 16 times; the third detection head 46 can be down-sampled by 32 times; and the fourth detection head 47 can be down-sampled by 64 times. Through the down-sampling of multiple detection heads with different multiples, the sampling accuracy of the features is improved, and the accuracy of the initial key points and the initial detection frame obtained by the subsequent key point branch and the detection branch is improved, and the accuracy of target detection is improved.

[0060] The detection parameter of the detection head in the embodiment is Conv2d (256, na * (nc + 5 + 3 * no_kpt), 1, 1). Conv2d is two-dimensional convolution, 256 is the number of channels, na represents the number of detection frame anchors, and na is 4 in the embodiment, and na can be another positive integer in other embodiments. no_kpt is the number of initial key points corresponding to each initial detection frame. Since the key points are represented by x, y, and conf, they are multiplied by 3, nc represents the category of target detection (nc >= 1), and 5 represents the center point coordinates, width, height, and confidence of the key frame, that is, x, y, w, h, and conf. The final detection sub-network 43 obtains no_kpt initial key points corresponding to each initial detection frame.

[0061] The correspondence between the key points and the detection frame in the application can be obtained based on the specific meaning of the key points and the detection frame. For example, when the target object is a vehicle, the detection frame is the detection frame of the vehicle, and each key point on the vehicle naturally corresponds to the detection frame of the vehicle.

[0062] Specifically, in the target detection process, after the to-be-detected image is input into the target detection model 40, the to-be-detected image is first subjected to feature extraction by the feature extraction sub-network 41 to obtain a feature map of the to-be-detected image; then the feature map is input into the adjustment sub-network 42 for feature adjustment to obtain features of different scales, and the features of different scales are respectively input into each detection head of the detection sub-network 43 for feature detection to obtain multiple initial detection frames of the target object and initial key points corresponding to each initial detection frame.

[0063] The target detection model is trained based on the positions of the training detection frames of the training images, and the positions of the multiple training key points corresponding to the training detection frames are limited.

[0064] The embodiment adjusts the positions of the multiple training key points corresponding to the training detection frames based on the positions of the training detection frames of the training images, thereby constraining the prediction of the positions of the key points, making the positions learned by the key points related to the detection frame, and accelerating the learning efficiency of the positions of the key points and making the learning of the positions of the key points more reasonable around the detection frame. The specific training method of the target detection model is described in Figure 3 Embodiment.

[0065] Step S23: performing a de-duplication process on the multiple initial detection frames and the multiple initial key points to obtain a detection frame of the target object and multiple key points corresponding to the detection frame.

[0066] Since the multiple initial detection boxes and the multiple initial key points are obtained by different times of down-sampling of the same feature by the multiple detection heads, there are overlapping and repetition in the multiple initial detection boxes and the multiple initial key points, which is not conducive to the result of target detection.

[0067] The multiple initial detection boxes and the multiple initial key points are processed to filter out the redundant overlapping and repeated detection boxes and key points, to obtain the detection box of the target object with the optimal solution and the multiple key points corresponding to the detection box. When there are multiple target objects on the image to be detected, there can be multiple detection boxes with the optimal solution, but there is only one detection box with the optimal solution for one target object.

[0068] In a specific application scenario, the multiple initial detection boxes can be subjected to non maximum suppression (NMS) to filter out the redundant initial detection boxes, and the key points bound to the initial detection boxes are also filtered out, so as to obtain the detection box of the target object with the optimal solution and the multiple key points corresponding to the detection box.

[0069] In a specific application scenario, the localization box deduplication algorithm can also be used for deduplication processing. That is, each initial detection box is judged. If the label of a certain initial detection box is the same as that of another initial detection box, and the overlapping area exceeds a certain threshold, the maximum circumscribed rectangle of the two initial detection boxes is found, and the maximum circumscribed rectangle is used as a new detection box to replace the original two initial detection boxes, so that the two initial detection boxes are fused, and the same is repeated until the detection box of the target object with the optimal solution is obtained, and then the multiple key points corresponding to the detection box are obtained based on the corresponding relationship. The threshold can be 85%, 80%, 70% of the area itself, which is not limited here.

[0070] The embodiment simultaneously predicts the detection box of the target object and the multiple key points corresponding to the detection box, can provide global information around the target object by the detection box, and obtain the accurate position of the feature key point of the target object by the key points, so as to improve the information richness of the target detection result, and then be suitable for various target detection scenarios, and improve the applicability of the target detection method.

[0071] Please refer to Figure 5 , Figure 5 is a schematic diagram of an embodiment of a target detection result of an image to be detected. The embodiment takes a vehicle as a target object, and takes a scenario of detecting the vehicle ground key point as an example for description.

[0072] The whole vehicle is marked by a detection frame 50, and at the same time, the vehicle ground key points 51 of the vehicle are also marked, wherein the vehicle ground key points 51 can be connected according to the structure of the vehicle to improve the saliency of the vehicle ground key points 51.

[0073] Please refer to Figure 6 , Figure 6 is a schematic diagram of another embodiment of a target detection result of a to-be-detected image. This embodiment takes a vehicle as the target object and takes the scene of detecting the roof key points of the vehicle as an example for description.

[0074] The whole vehicle is marked by a detection frame 60, and at the same time, the roof key points 61 of the vehicle are also marked, wherein the roof key points 61 can be connected according to the structure of the vehicle to improve the saliency of the roof key points 61.

[0075] In other scenes, the key points can also be selected as other types of key points of the vehicle, such as lamp key points, window key points, etc., which are set based on actual needs.

[0076] The above target detection result includes the detection frame of the target object and the plurality of key points corresponding to the detection frame, which can improve the information richness of the target detection result, and is suitable for a variety of target detection scenes.

[0077] Through the above steps, the target detection method of the embodiment detects the to-be-detected image through the target detection model to obtain the detection frame of the target object and the plurality of key points corresponding to the detection frame, which can improve the information richness of the target detection result, and is suitable for a variety of target detection scenes, thereby improving the applicability of the target detection method. The target detection model includes a feature extraction subnetwork, an adjustment subnetwork and a detection subnetwork which are cascaded with each other, wherein the detection subnetwork includes a plurality of detection heads to perform different times of down-sampling on the adjusted features to improve the sampling accuracy of the features, thereby improving the accuracy of the initial key points and the initial detection frame obtained by the subsequent key point branch and the detection branch, and improving the accuracy of the target detection. Finally, the plurality of initial detection frames and the plurality of initial key points are de-duplicated to obtain the detection frame of the target object and the plurality of key points corresponding to the detection frame, which improves the pertinence of the target detection result to the optimal solution, avoids repeated or invalid results, and is beneficial to the clear presentation of the target detection result.

[0078] Please refer to Figure 7 , Figure 7 is a training flowchart of an embodiment of the target detection model of the application.

[0079] Step S71: obtaining a training image.

[0080] The type of the training image is determined based on the target object of the target detection model. For example, when the target object is a vehicle, the training image is an image containing a vehicle. When the target object is a human face, the training image is an image containing a human face.

[0081] During the training process, multiple training images can be used for model training.

[0082] Step S72: The target detection model in the training process is used to perform target detection on the training image to obtain multiple training bounding boxes and multiple training key points corresponding to each training bounding box.

[0083] The structure of the target detection model of the embodiment can be the same as that of any target detection model in the embodiments, and details are described above. Figure 2 The structure of the target detection model of the embodiment can be the same as that of any target detection model in the embodiments, and details are described above.

[0084] In a specific application scenario, the target detection model in the training process is used to perform target detection on the training image through the feature extraction subnetwork, the adjustment subnetwork, and the detection subnetwork of the target detection model in the training process in sequence to obtain multiple training bounding boxes of the training image and training key points corresponding to each training bounding box. The specific target detection process is the same as that of the embodiments, and details are described above. Figure 2 The specific target detection process is the same as that of the embodiments, and details are described above.

[0085] Step S73: The position of the training key point is limited based on the training bounding box.

[0086] After obtaining the multiple training bounding boxes of the training image and the training key points corresponding to each training bounding box, the position of the training key point is limited based on the training bounding box. Since the embodiment is a training process, after obtaining the multiple training bounding boxes, it is not necessary to perform a de-duplication process on the training bounding boxes, and the training is directly performed based on the multiple training bounding boxes and the training key points corresponding to each training bounding box to increase the number of training samples of the model and facilitate model learning.

[0087] In a specific application scenario, the position of the training key point can be limited within a preset range of the training bounding box by using a compression function. The preset range of the training bounding box can be determined based on the center point of the training bounding box and the sum of the width and height of the training bounding box with the first preset proportion. The compression function can be a sig function or other compression functions, which are not limited herein.

[0088] In a specific application scenario, the position of the training key point limited by the sig function can be calculated by using the following formula:

[0089]

[0090] Kpt x =sig(pred x)*(thres1-thres2)*anchor_grid x +grid x *stride_x (2)

[0091] Kpt y =sig(pred y )*(thres1-thres2)*anchor_grid y +grid y *stride_y (3)

[0092] Wherein, the formula (1) is the calculation method of sig function. Wherein pred x is the output of the target detection model, which is the horizontal coordinate of the training key point predicted by the model, pred y is the vertical coordinate of the training key point predicted by the target detection model, stride_x and stride_y represent the step length of the target detection model in the width and height aspect, grid x and grid y are the coordinates of the grid where the center point of the training detection frame anchor is located. Kpt x and Kpt y are the horizontal and vertical coordinates of the training key point after the sig function is limited. Anchor_grid x and anchor_grid y are the width and height of the training detection frame anchor.

[0093] In this application scenario, first, the sig function is used to limit the range of pred x and pred y predicted by the target detection model to [0, 1], and the position of the training key point is limited to the position of the center point of the corresponding training detection frame anchor+the width and height of the training detection frame*the first preset proportion by using formula (2) and formula (3), wherein the first preset proportion is [-thres1, thres1-thres2].

[0094] In this application scenario, the value of thres2 can be set to 2 and the value of thres1 can be set to 4, so that the key points can be detected outside the frame, and the key point position can also be better constrained to prevent accuracy from being inaccurate. In other application scenarios, the values of thres2 and thres1 can also use other empirical proportions, which are not limited here.

[0095] In other application scenarios, the manner of limiting the positions of the training key points within the preset range of the training detection box can also use setting a range threshold for division adjustment or other compression functions, and the specific implementation is not limited here.

[0096] By limiting the positions of the training key points within the preset range of the training detection box through the above method, the learning of the key point positions by the target detection model can be accelerated, and the learning of the key point positions can be more reasonable, thereby improving the model training precision.

[0097] Step S74: training the target detection model in training by using the limited training key points to obtain a trained target detection model.

[0098] A plurality of standard key points are obtained; wherein the standard key points can be standard data of key points obtained by manual labeling based on training images.

[0099] The extended width and height range is determined based on the product of the width and height of the training detection box and the second preset proportion. The second preset proportion can be set based on actual conditions, for example, 0.05, 0.06, 0.08, etc.

[0100] Based on the extended width and height range, the limited training key points are extended with each limited training key point as the center to obtain a plurality of training key boxes; and based on the extended width and height range, the standard key points are extended with each standard key point as the center to obtain a plurality of standard key boxes.

[0101] The sizes of the training key boxes and the standard key boxes are the same as the extended width and height range.

[0102] The target detection model in training is trained by using the gap between each training key box and the corresponding standard key box to obtain a trained target detection model.

[0103] Please refer to Figure 8 , Figure 8 is a schematic view of an embodiment of a training key box and a corresponding standard key box.

[0104] The training detection box 80 corresponds to a plurality of training key points 84. Each training key point 84 has a corresponding manually labeled standard key point 83.

[0105] In one specific application scenario, when the width and height of the training detection box are box_w and box_h respectively, the extended width and height range are w=box_w*thresh and h=box_h*thresh respectively. Wherein, thresh is the second preset proportion.

[0106] Based on the extended width and height range, the training key points 84 are centered to expand the limited training key points to obtain a plurality of training key frames 81; based on the extended width and height range, the standard key points 83 are centered to expand the standard key points 83 to obtain a plurality of standard key frames 82.

[0107] The target detection model in training is trained by the loss function using the angle (Angle cost) gap, distance (distance cost) gap, shape (shape cost) gap and intersection over union (iou) between the training key frame 81 and the standard key frame 82, to obtain the trained target detection model.

[0108] Since the embodiment uses the gap between the frames to train the model, the training parameter degree can be expanded from the distance difference between the points to the angle difference, distance difference, shape difference and intersection over union, thereby enriching the training parameters, improving the training difficulty, and further improving the training effect of the model.

[0109] In a specific application scenario, Siou loss function can be used to help training. Among them,

[0110] The Siou loss function can improve the detection frame learning ability, which is composed of angle (Angle cost) loss, distance (distance cost) loss, shape (shape cost) loss and iou loss. The introduction of angle loss in Siou loss function is mainly to assist the distance calculation between two frames. Because most of the predicted frames are not intersected with the real frames at the beginning of training, the introduction of angle loss can accelerate convergence. Distance loss is constructed by the distance between the centers of the two frames and the long frame of the circumscribed rectangle, but there is an additional item γ. This item is related to the angle between the above two frames. Shape loss is the aspect ratio between the two frames, which is defined by calculating the difference between the width of the two frames and the ratio of the maximum width or height between them. Iou loss is the intersection over union loss. The loss function of Siou is as follows:

[0111]

[0112] Among them, IoU is the intersection over union loss, Δ represents the distance loss, and Ω represents the shape loss.

[0113] In a specific application scenario, the total loss function can be as follows:

[0114] L all =l1*L siou +l2*L cls +l3*L obj +l4*L kpts_iou (5)

[0115] wherein, L kpts_iou represents the key point loss; L cls , L obj is a conventional loss function. l1, l2, l3, and l4 are respectively balance coefficients between various loss functions, which need to be adjusted according to data distribution and specific loss.

[0116] In this embodiment, the loss function containing the angle loss, the distance loss, the shape loss, and the IoU is used to train the target detection model in the training, which can enrich the training parameters, improve the training difficulty, and further improve the training effect of the model.

[0117] In other application scenarios, when other loss functions can also contain the angle loss, the distance loss, the shape loss, and the IoU loss function or even more losses, other loss functions can also be used to train the target detection model.

[0118] wherein, the loss of the training bounding box can be trained by using a conventional loss function. No limitation is made herein.

[0119] In a specific application scenario, the target detection method of this embodiment is applied to image recognition. After the target detection model is used to perform target detection on the to-be-detected image, the detection bounding box of the target object and the plurality of key points corresponding to the detection bounding box are obtained, the target object can be recognized based on the detection bounding box of the target object and the plurality of key points corresponding to the detection bounding box, and the type of the target object is obtained.

[0120] In a specific application scenario, the target detection method of this embodiment is applied to image classification. After the target detection model is used to perform target detection on the to-be-detected image, the detection bounding box of the target object and the plurality of key points corresponding to the detection bounding box are obtained, the target object can be classified based on the detection bounding box of the target object and the plurality of key points corresponding to the detection bounding box, and the category of the target object is obtained.

[0121] In other application scenarios, the target detection method of this embodiment can also be applied to other image processing scenarios, which are not limited herein.

[0122] Through the above steps, the training of the target detection model of this embodiment first limits the position of each training key point relative to the center point of the corresponding training detection bounding box within a certain range, secondly, each training key point is expanded into a rectangular target box according to a certain proportion, and finally the siou is used as the key point loss function for the expanded rectangular standard key point and the expanded rectangular predicted training key point, so that the position of the predicted key point is more accurate through the coincidence degree of the key point expanded box, better detection effect and detection precision are obtained, and the method can be applied to various complex scenarios.

[0123] Based on the same inventive concept, the present application further provides an electronic device capable of being executed to implement the target detection method of any of the above embodiments, please refer to Figure 9 , Figure 9 is a structural schematic diagram of an embodiment of the electronic device provided by the present application, and the electronic device comprises a processor 91 and a memory 92.

[0124] The processor 91 is used to execute the program instructions stored in the memory 92 to implement the steps of any of the above target detection methods. In a specific implementation scenario, the electronic device can include but is not limited to: a microcomputer, a server, in addition, the electronic device can also include a notebook computer, a tablet computer and other mobile devices, which are not limited here.

[0125] Specifically, the processor 91 is used to control itself and the memory 92 to implement the steps of any of the above embodiments. The processor 91 can also be called a processor (Central Processing Unit, CPU). The processor 91 can be an integrated circuit chip with signal processing capability. The processor 91 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 91 can be realized by integrated circuit chips together.

[0126] The above scheme improves the target detection accuracy of the target detection model.

[0127] Based on the same inventive concept, the present application further provides a computer readable storage medium, please refer to Figure 10 , Figure 10 is a structural schematic diagram of an embodiment of the computer readable storage medium provided by the present application. The computer readable storage medium 100 stores at least one program data 101, and the program data 101 is used to implement any of the above methods. In an embodiment, the computer readable storage medium 100 includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk and various storage program codes. Medium.

[0128] In several embodiments provided by the present application, it should be understood that the disclosed method and device can be implemented in other manners. For example, the described device embodiments are merely schematic, and the division of the modules or units can be different, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0129] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0130] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or directly contribute to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium.

[0132] The above descriptions are merely some embodiments of the present application, and thus do not limit the patent scope of the present application. Any equivalent structure or equivalent flow transformation made by using the contents of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target detection method, characterized in that, include: Obtain the image to be detected, which includes the target object; The target detection model is used to detect the target in the image to be detected, and the detection box of the target object and multiple key points corresponding to the detection box are obtained. The target detection model is obtained by restricting the positions of multiple training keypoints based on the positions of the training detection boxes in the training image, thereby constraining the prediction of keypoint positions based on the positions of the training detection boxes. This ensures that the learned positions of the keypoints are related to and restricted to the vicinity of the detection boxes.

2. The target detection method according to claim 1, characterized in that, The target detection model includes a cascaded feature extraction subnetwork, an adjustment subnetwork, and a detection subnetwork. The step involves performing target detection on the image to be detected using a target detection model to obtain the detection bounding box of the target object and multiple key points corresponding to the detection bounding box, including: The target detection is performed on the image to be detected sequentially through the feature extraction subnetwork, the adjustment subnetwork and the detection subnetwork to obtain multiple initial detection boxes of the target object and initial key points corresponding to each initial detection box; The initial detection boxes and the initial key points are deduplicated to obtain the detection box of the target object and the corresponding key points.

3. The target detection method according to claim 2, characterized in that, The detection subnetwork includes multiple detection heads, each of which is connected to the adjustment subnetwork. The step of sequentially performing target detection on the image to be detected through the feature extraction subnetwork, the adjustment subnetwork, and the detection subnetwork to obtain multiple initial detection boxes of the target object and initial key points corresponding to each initial detection box includes: The feature extraction sub-network is used to extract features from the image to be detected, resulting in a feature map of the image to be detected. The feature map is input into the adjustment sub-network for feature adjustment to obtain features at multiple different scales; Multiple features of different scales are input into the corresponding detection heads for detection processing at different magnifications, resulting in multiple initial detection boxes and initial key points corresponding to each initial detection box.

4. The target detection method according to claim 1, characterized in that, The process of acquiring the image to be detected, which includes the target object, includes: The training images were obtained; The training image is used to detect objects by the target detection model in training, and multiple training detection boxes and multiple training key points corresponding to each training detection box are obtained. The positions of the training key points are restricted based on the training detection box; The target detection model is trained using the restricted training keypoints to obtain the trained target detection model.

5. The target detection method according to claim 4, characterized in that, The restriction of the position of the training keypoints based on the training detection box includes: The location of the training key points is restricted to a preset range within the training detection box by a compression function.

6. The target detection method according to claim 5, characterized in that, The step of restricting the position of the training key points to a preset range within the training detection box using a compression function further includes: The preset range of the training detection box is determined based on the sum of the width and height of the training detection box at a first preset ratio and the center point of the training detection box.

7. The target detection method according to claim 4, characterized in that, The step of training the target detection model using the restricted training key points to obtain the trained target detection model includes: Multiple standard key points were obtained; The extended width and height range is determined based on the product of the width and height of the training detection box and the second preset ratio. Based on the expanded width and height range, with each restricted training key point as the center, the restricted training key points are expanded to obtain multiple training key boxes. Based on the extended width and height range, each standard key point is expanded to obtain multiple standard key frames by taking each standard key point as the center. The target detection model is trained by utilizing the difference between each training keyframe and the corresponding standard keyframe, thus obtaining the trained target detection model.

8. The target detection method according to claim 7, characterized in that, The step of training the target detection model using the difference between each training keyframe and the corresponding standard keyframe to obtain the trained target detection model includes: The target detection model is trained by using the angle difference, distance difference, shape difference and intersection-union comparison between the trained keybox and the standard keybox through the loss function, and the trained target detection model is obtained.

9. The target detection method according to claim 1, characterized in that, The target detection method is applied to image recognition; The step involves performing target detection on the image to be detected using a target detection model to obtain the detection bounding box of the target object and multiple key points corresponding to the detection bounding box, followed by: The target object is identified based on the detection bounding box of the target object and multiple key points corresponding to the detection bounding box, thereby obtaining the type of the target object.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the target detection method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program data that can be executed to implement the target detection method as described in any one of claims 1 to 9.

Images