Neural network training method, target detection method, device, medium and product

Abstract

The application discloses a neural network training method, a target detection method, equipment, a medium and a product, and relates to the technical field of computers. The method comprises the following steps: receiving a verification picture set collected from a target detection scene, and generating feature maps of pictures in the verification picture set, wherein the data types labeled by the pictures comprise a detection frame and a plurality of key points, a plurality of predicted key points of each feature map and key point scores are calculated; obtaining a key point regression result after determining that the feature map has the predicted key points according to labeled data and the key point scores; and adjusting structure parameters of a neural network according to an image detection result obtained from the key point regression result and the labeled data. The application learns picture data in the verification picture set, which is labeled with the detection frame and the key points at the same time, by training the neural network, to determine whether the predicted key points of the neural network are reliable, and to ensure the improvement of the detection effect in the current detection task framework.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a neural network training method, object detection method, device, medium, and product. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. It should not be construed as an admission that the description herein is prior art.

[0003] In urban and industrial scenarios, traditional object detection algorithms based on bounding boxes can no longer meet the needs of complex and precise tasks. Moreover, these traditional object detection algorithms can only provide the two-dimensional coordinates of the bounding box of the object in the pixel coordinate system or camera coordinate system. They cannot obtain whether the detected target is placed on a certain plane, nor can they obtain the pose and orientation of the detected target. In practical applications, this information is crucial and is even the core requirement for the algorithm to be implemented in the service scenario—such as the overlap judgment of vehicles and parking spaces, and the 3D coordinates of workpieces in the world coordinate system.

[0004] To meet the demands of complex and precise target detection, some intuitive solutions involve using devices such as LiDAR, binocular cameras, and ToF cameras (Time of Flight) to acquire depth information in addition to 2D data. This combined 2D and depth information is then used to determine the object's position and orientation in the real-world coordinate system. However, in practical IoT applications, solutions incorporating these depth acquisition devices are prohibitively expensive, significantly reducing product competitiveness. Furthermore, existing customers with pure vision-based urban and industrial solutions cannot upgrade their software. Current target detection algorithms based on bounding boxes and keypoints, designed to meet complex and precise detection needs, require labeling all images with keypoints, incurring substantial labor and financial costs. Without keypoint labeling, the target detection neural network is prone to misjudging and discarding image data containing actual keypoints. For example, in practical 3D detection applications, occlusion cases often present stable bounding boxes but lack complete keypoints. When encountering numerous occluded cases, the target detection neural network cannot provide stable perceptual results for subsequent decision-making algorithms.

[0005] It is evident that there is an urgent need to provide a solution for the core algorithm of target detection that can meet the requirements of complex and refined detection tasks while also satisfying the 2D detection needs of existing customers. Summary of the Invention

[0006] This application provides a neural network training method, object detection method, device, medium, and product to at least solve the problem in the prior art that the labor cost and capital required to annotate key points on all images are very high.

[0007] According to one aspect of this application, a neural network training method is also provided, comprising:

[0008] Receive a set of verification images with labeled data collected from a target detection scene, and generate feature maps for each image in the set of verification images. The data types labeled by the labeled data include detection boxes and multiple key points.

[0009] Calculate multiple predicted key points for each of the feature maps, and the key point score for each predicted key point;

[0010] Based on the labeled data and the key point scores, determine the probability that the predicted key point is located at the center point of the feature map;

[0011] When it is determined from the probability that the predicted key point is located at the center point, the predicted key point is subjected to key point regression processing based on the marked key point to obtain the key point regression result. The key point regression result is used to indicate the offset of the predicted key point relative to the center point.

[0012] The feature map is processed based on the key point regression results to obtain the image detection results for each image, and the structural parameters of the neural network are adjusted based on the image detection results and the labeled data.

[0013] In some embodiments, the step of determining the probability that the predicted keypoint is located at the center point of the feature map, based on the labeled data and the keypoint score, includes:

[0014] A loss function is calculated for each type of labeled data and the keypoint score to determine whether the predicted keypoint is located at the center point position based on the loss function.

[0015] In some embodiments, before determining the probability that the predicted keypoint is located at the center point of the feature map based on the labeled data and the keypoint scores, the method further includes:

[0016] Obtain the position of the first center point determined when labeling the detection box;

[0017] The step of determining the probability that the predicted keypoint exists at the center point of the feature map based on the labeled data and the keypoint score includes:

[0018] A loss function is calculated on the first center point location and the key point score to determine whether the predicted key point exists at the center point location based on the loss function.

[0019] In some embodiments, the step of calculating a loss function on the first center point location and the keypoint score to determine whether the predicted keypoint is present at the center point location according to the loss function includes:

[0020] With minimizing the loss function as the calculation objective, when the difference between the first center point location and the key point score is less than a preset threshold, it is determined that the predicted key point exists at the center point location.

[0021] In some embodiments, the step of performing keypoint regression processing on the predicted keypoints based on the labeled keypoints includes:

[0022] The regression result of the key points is determined by minimizing the distance between the predicted key points and the labeled key points.

[0023] In some embodiments, the step of performing keypoint regression processing on the predicted keypoints based on the labeled keypoints further includes:

[0024] The first regression objective is to minimize the distance between the predicted keypoints and the labeled keypoints. The second regression objective is to ensure that the distance between the predicted keypoints meets a predetermined distance range. The regression result of the keypoints is determined based on the first regression objective and the second regression objective.

[0025] In some embodiments, before determining the probability that the predicted keypoint is located at the center point of the feature map based on the labeled data and the keypoint scores, the method further includes:

[0026] Based on a preset score threshold range, the scores of the key points within the score threshold range are filtered out, so as to determine the probability that the predicted key point is located at the center point of the feature map based on the labeled data and the filtered key point scores.

[0027] In some embodiments, before determining the probability that the predicted keypoint is located at the center point of the feature map based on the labeled data and the keypoint scores, the method further includes:

[0028] Obtain the prediction bounding box determined based on the predicted key points, and determine the degree of overlap between the prediction bounding box and the labeled detection bounding box. Based on the degree of overlap, filter out the images with positive sample detection bounding boxes.

[0029] According to another aspect of this application, a target detection method is also provided, the method comprising:

[0030] Receive a set of images to be detected with labeled data collected from a target detection scene, and generate feature maps of each image to be detected in the set of images to be detected, wherein the data type labeled by the labeled data includes at least detection boxes;

[0031] The feature maps of each image are processed using a neural network to obtain the image detection result of the image to be detected, wherein the neural network is trained using the neural network training method described above.

[0032] According to another aspect of the application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the neural network training method.

[0033] According to another aspect of this application, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the neural network training method described above.

[0034] According to another aspect of this application, a computer program product is also provided, the computer program product comprising a computer program that, when executed by a processor, implements the steps of the neural network training method.

[0035] In this embodiment, a neural network is trained to learn from image data in a verification image set that simultaneously annotates detection boxes and keypoints. This allows the reliability of the keypoints predicted by the neural network to be determined. Before using the neural network for object detection, it is unnecessary to annotate all images with keypoints. The neural network directly uses keypoint scores to estimate whether images without annotated keypoint information can output reliable keypoints, rather than directly rejecting the possibility that images without annotated keypoint information contain keypoints. Furthermore, this application can achieve full-scale keypoint learning on existing 2D detection annotation datasets by adding a small number of keypoint annotations, ensuring improved detection performance within the current detection task framework without requiring customers to purchase new computing hardware and camera equipment. Attached Figure Description

[0036] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0037] Figure 1 This is a flowchart of a neural network training method according to an embodiment of this application;

[0038] Figure 2 This is a flowchart of a target detection method according to an embodiment of this application;

[0039] Figure 3 This is a multi-task neural network structure diagram according to another embodiment of this application. Detailed Implementation

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0042] Current object detection neural network training processes based on bounding boxes predict categories, bounding boxes, and foreground scores. However, traditional bounding box-based object detection results are insufficient for complex and detailed detection tasks. To meet these needs, current solutions utilize 2D and depth information to determine the object's position and pose in the real-world coordinate system. However, introducing depth acquisition equipment into these applications leads to high product prices, significantly reducing product competitiveness, and hinders software upgrades for existing urban and industrial pure vision solutions. Furthermore, current bounding box-plus-keypoint object detection algorithms that meet complex and detailed detection requirements require annotating keypoints on all images, incurring very high labor and financial costs.

[0043] To address the aforementioned problems, the first embodiment of this invention proposes a neural network training method. When using this method to train a neural network for target detection, it is compatible with the existing incremental monocular 3D detection algorithm with its own detection boxes. Simultaneously, it accurately outputs both 2D and 3D detection boxes, without significantly increasing the computational load. First, please refer to... Figure 1 The neural network training method includes the following steps:

[0044] Step S11: Receive the set of verification images with labeled data collected from the target detection scene, and generate feature maps (feature_map) for each image in the verification image set. The data types of the labeled data include detection boxes and multiple key points.

[0045] Step S12: Calculate multiple predicted keypoints for each feature map, and the keypoint scores for each predicted keypoint. For example, the keypoint scores calculated by the neural network for a feature map may be 0.998, 0.997, etc.

[0046] Step S13: Based on the labeled data and key point scores, determine the probability that the center point of the feature map has a predicted key point.

[0047] Step S14: When it is determined from the probability that there is a predicted key point at the center point position, perform key point regression processing on the predicted key point based on the marked key point to obtain the key point regression result. The key point regression result is used to indicate the offset of the predicted key point relative to the center point position.

[0048] Step S15: Perform image processing on the feature map based on the keypoint regression results to obtain the image detection results for each image. Adjust the structural parameters of the neural network based on the image detection results and the labeled data. As a result, the neural network with adjusted structural parameters can predict reliable keypoints even for feature maps with only labeled detection boxes. Thus, while outputting 2D box detection results based on the labeled detection boxes, it can also output 3D box detection results based on the predicted keypoints, meeting complex and detailed detection requirements while reducing labor costs and investment.

[0049] When a neural network processes an image, it determines the offset of the predicted keypoint from the center point of the feature map based on the keypoint regression results. This offset is then used to obtain the image detection result. For example, in an industrial scene, the image detection result is the output of the 3D bounding box of the workpiece; in a roadside parking scene, the image detection result is the car tracking bounding box. In other words, keypoint regression indicates the offset of the learned predicted keypoint relative to the center point (x, y) of the feature map. The final predicted keypoint learned by the neural network is N values ​​obtained by offsetting the center point (x, y) according to the keypoint regression results, where N is a positive integer, and its specific value depends on the object being detected. For example, when detecting cars, N is eight, meaning there are eight key points. In the license plate detection task, N is four. The license plate key point regression task can be written as (x1, y1, x2, y2, x3, y3, x4, y4), xi = x + delta(x), where delta(x) is the key point regression result of the corresponding predicted key points learned by the neural network.

[0050] Step S13, which involves determining the probability of a predicted keypoint at the center point of the feature map based on the labeled data and keypoint scores, includes:

[0051] A loss function is calculated for each type of labeled data and keypoint score to determine whether a predicted keypoint exists at the center point location. Specifically, a loss function is calculated for the labeled detection boxes and their corresponding keypoint scores, thereby training the neural network to find the keypoint that should exist at the center point location of the feature map based on the labeled detection box data. A loss function is also calculated for the labeled keypoints and their scores, training the neural network to learn the keypoints present at the center point location of the feature map based on the labeled data with keypoints. For images without labeled detection boxes or keypoints, the neural network does not need to learn.

[0052] During the specific training process, when the neural network determines the probability that a keypoint is predicted at the center point of the feature map, this probability can be calculated using the cross-entropy loss function `cross entropy(predict, gt)`. In traditional cross-entropy loss functions, the ground truth `gt` represents whether a keypoint is labeled at a certain location. However, the ground truth `gt` provided in this embodiment divides the labeled data into three parts: 1) both detection boxes and keypoints are labeled. 2) only detection boxes are labeled. 3) neither detection boxes nor keypoints are labeled. In this embodiment, images without labeled detection boxes and keypoints are considered background images, while images with labeled detection boxes and keypoints are considered foreground images. After the neural network calculates the keypoint score of the feature map with only labeled detection boxes, it automatically determines whether the image with only labeled detection boxes is a foreground or background image when calculating the cross-entropy loss function for each type of labeled data and keypoint score. This filters out occlusion cases and outputs stable perceptual results for the subsequent regression processing of the neural network.

[0053] When calculating the overall loss using the cross-entropy(predict, gt) function for each labeled data and keypoint score, the calculated cross-entropy overall loss function (landmark Label Assigner) is expressed as follows:

[0054] Loss lmk_obj_total =Loss lmk_obj +Loss lmk_pos +Loss lmk_neg .

[0055] Overall Loss lmk_obi_total middle:

[0056] Losslmk_obj =CrossEntropy(lmkobj pred *lmkobj gt lmkobj gt ), where lmkobj pred lmkobj is used to score key points in the output of a neural network. gt This refers to the center points of multiple key points in a manually labeled feature map. (Loss) lmk_obj The calculation of the cross-entropy loss function is used to train the neural network to learn the corresponding key points based on the labeled data, that is, to ensure that the neural network can learn how to label key points based on the labeled key points.

[0057] Loss lmk_pos =CrossEntropy(lmkobj pred *mask pos obj gt *mask pos ), where mask pos =where(obj) gt -lmkobj pred <0.005), obj gt This represents the location of the center point of the 2D bounding box (i.e., the labeled detection box) in the feature map (i.e., the first center point location mentioned below). lmk_pos The calculation of the cross-entropy loss function is used to: train the neural network to find the corresponding predicted key points when the distance between the position of the center point of the 2D bounding box in the feature map and the key point score meets the set threshold, and determine that the feature map of the labeled detection box has key points.

[0058] Loss lmk_neg =CrossEntropy(lmkobj pred *mask neg obj gt *mask neg ), where mask neg =where(obj) gt =0), Loss lmk_negThe cross-entropy loss function is used to train the neural network to learn feature maps (feature maps) that have neither labeled detection boxes nor labeled keypoints without requiring training. Images in these feature maps are then directly identified as background images. Therefore, this embodiment of the invention can filter out a large number of occlusion cases through keypoint scores, providing a stable image perception result for the subsequent keypoint regression decision-making process.

[0059] Regarding Loss lmk_pos The calculation steps of the cross-entropy loss function specifically include, before determining the probability of a predicted keypoint at the center point of the feature map based on the labeled data and keypoint scores in step S13, obtaining the first center point position determined during the annotation detection box. Step S13, determining the probability of a predicted keypoint at the center point of the feature map (feature_map) based on the labeled data and keypoint scores, specifically includes: calculating a loss function on the first center point position and keypoint scores to determine whether a predicted keypoint exists at the center point position based on the loss function. Specifically, when calculating the loss function, minimizing the loss function is the calculation objective. A predicted keypoint is determined to exist at the center point position when the difference between the first center point position and the keypoint score is less than a pre-set threshold. For example, if the difference between the first center point position and the keypoint score is less than 0.005, the neural network determines that a predicted keypoint exists at the center point of the feature map (feature_map). When calculating the loss function on the keypoint score at the first center point position, the Loss mentioned in the overall loss calculation of the cross-entropy loss function is used. lmk_pos The cross-entropy loss function is used to calculate the key points, and the result is used to determine whether the feature map of the labeled detection box has stable key point outputs, rather than directly denying the possibility that the feature map of the labeled detection box has key points. Therefore, when using the trained neural network to perform object detection on monocular images, it can accurately output both 2D and 3D detection boxes simultaneously, without a significant increase in computational cost.

[0060] This invention solves the problem that existing target detection methods based on keypoint regression cannot evaluate keypoint quality by calculating the probability of a predicted keypoint at the center point location using the cross-entropy loss function. Addressing the numerous occlusion cases encountered in practical 3D detection applications, where stable bounding boxes exist but lack complete keypoints, the neural network trained according to this invention filters out a large number of occlusion cases by predicting the probability of keypoint existence based on keypoint scores during target detection. This results in stable perceptual outputs for subsequent detection by the neural network.

[0061] Step S14 involves performing keypoint regression processing on the predicted keypoints based on the labeled keypoints.

[0062] The regression objective is to minimize the distance between predicted and labeled keypoints to determine the keypoint regression results. To further improve the regression performance, the regression process uses minimizing the distance between predicted and labeled keypoints as the first regression objective, and simultaneously ensuring that the distance between predicted keypoints meets a predetermined range as the second regression objective. The keypoint regression results are determined based on the first and second regression objectives. If the first regression objective is calculated using the Wing loss function and the second regression objective is calculated using the bone loss function, and both Wing loss and bone loss functions are used for keypoint regression processing, the keypoint regression task can be specifically expressed by the following formula:

[0063] Loss lmk_reg =wing loss(lmk) pred ,lmk gt )+0.1*bone loss(lmk pred ,lmk gt ).

[0064] In the formula, lmk pred lmk is the location of the predicted keypoints in the neural network's prediction output. gt This refers to the positions of multiple manually labeled keypoints in the image. The purpose of the Wing loss function is to minimize the distance between the predicted keypoints output by the neural network and the labeled keypoints. Specifically, the distances between predicted keypoints must meet a predetermined range. This is achieved by using the bone loss function to ensure that the distribution of distances between predicted keypoints is consistent with the distribution of pairwise distances between labeled keypoints. In this embodiment of the invention, when both the Wing loss function and the bone loss function are used for keypoint regression calculation, the regression results are more accurate.

[0065] In this embodiment of the invention, during the training of the neural network, when setting an average precision (AP) metric based on keypoints, it is observed that the AP metric continuously increases on the validation image set (images in the validation image set are labeled with both bounding boxes and keypoints). Consequently, the regression quality obtained when the neural network performs keypoint regression processing also improves accordingly. Simultaneously, in the foreground image portion, foreground images with positive bounding boxes are determined by the intersection-over-union (IOU) ratio between the predicted keypoints and the ground truth values ​​estimated by the network. Therefore, when calculating the regression loss function, bounding boxes with better regression quality are more likely to be assigned to the foreground.

[0066] To ensure that the comprehensive solution pipeline for neural networks in object detection becomes clearer and simpler, before step S13 determines the probability that a predicted key point exists at the center point of the feature map based on the labeled data and key point scores, the method further includes: filtering out key point scores within a preset score threshold range, so as to determine the probability that a predicted key point exists at the center point of the feature map based on the labeled data and the filtered key point scores.

[0067] Current heatmap-based object detection methods output a score for each keypoint. In contrast, the neural network trained in this invention only needs to determine the quality of keypoint predictions for an object, without needing to determine if every keypoint is predicted correctly. In code, this means the neural network should ideally output only one keypoint score for each object's keypoints. Based on this score, the neural network indicates whether the keypoint should be used for subsequent keypoint reliability prediction and keypoint regression (keypoint reliability prediction determines whether the feature map has the probability of predicting keypoints). For example, by setting a single threshold to reject unqualified keypoint scores, given that keypoint scores range from zero to one, keypoints with scores below 0.995 are considered unqualified. This prevents the neural network from using unqualified keypoints for reliable keypoint judgment and subsequent keypoint regression calculations, making the entire neural network solution pipeline clear and simple. Therefore, when using the neural network trained by the embodiments of the present invention for target detection, the neural network will only output a key point score. This reduces the amount of computation while also taking into account the actual needs of finer-grained detection tasks. That is, the neural network trained by the embodiments of the present invention does not need to determine the quality of each key point, but rather whether the current object has stable key point outputs.

[0068] Before step S13, which determines the probability that a predicted keypoint exists at the center point of the feature map based on the labeled data and keypoint scores, the neural network training method provided in this embodiment of the invention further includes:

[0069] The process involves obtaining predicted bounding boxes based on predicted keypoints and determining the degree of overlap between these predicted boxes and the labeled detection boxes. Images with positive detection boxes are then selected based on this overlap. Specifically, the neural network calculates the Intersection Over Union (IOU) ratio between its predicted bounding boxes and the labeled detection boxes to determine the degree of overlap between them and the ground truth detection boxes, thus selecting feature maps with positive detection boxes. When calculating the IOU, the predicted and detection boxes should overlap as much as possible; a higher degree of overlap indicates a greater likelihood that the corresponding feature map contains positive samples. After finding reliable predicted keypoints by calculating the overall loss using the labeled data with positive detection boxes and performing keypoint regression on these predicted keypoints, the neural network avoids wasting computational resources on negative samples, thus improving its detection efficiency in object detection.

[0070] Therefore, in this embodiment of the invention, positive samples are selected using detection boxes for the loss function calculation elements of the keypoint regression part during training. That is, during training, the neural network determines whether a location contains a positive sample using the intersection-over-union ratio (IoU) between predicted and detected boxes. The neural network only calculates the keypoint regression loss function on images with positive samples, and this loss function includes an implicit design for rigid body keypoints. This implicit design means that, without explicitly defining what a keypoint is, the neural network uses the loss function from keypoint regression processing to specify the features that keypoints must meet. In other words, the loss function is used to train the predicted keypoints estimated by the neural network, allowing the neural network to automatically select keypoints that meet the requirements of the loss function from keypoint regression processing. All predicted keypoints need to be sufficiently close to the labeled keypoints, and the pairwise distances between these predicted keypoints need to be consistent with the pairwise distance distribution of the labeled keypoints.

[0071] This invention trains a neural network to learn from image data in a dataset that simultaneously annotates detection boxes and keypoints. This allows the network to determine the reliability of predicted keypoints. Before using the neural network for object detection, it eliminates the need to annotate all images with keypoints. The neural network directly uses keypoint scores to estimate whether images without labeled keypoints can output reliable keypoints, rather than directly rejecting the possibility of keypoints in unannotated images. Furthermore, the neural network can learn keypoints from existing 2D detection and annotation datasets by adding a small number of keypoint annotations, ensuring improved detection performance within the current detection task framework without requiring customers to purchase new computing hardware or camera equipment.

[0072] In a second embodiment of the present invention, a target detection method is also provided, please refer to [link to relevant documentation]. Figure 2 The target detection method includes the following steps:

[0073] Step S21: Receive the set of images to be detected, labeled with annotated data, collected from the target detection scene, and generate feature maps (feature_maps) for each image in the set. The labeled data must include at least bounding boxes. In the set of images to be detected, some images may only have bounding boxes labeled, while others may have both bounding boxes and keypoints labeled. To facilitate differentiation, images with only bounding boxes are used as the comparison set, and images with both bounding boxes and keypoints are used as the verification set. The number of images in the comparison set is significantly greater than the number of images in the verification set. For example, if the proportion of images in the comparison set is much greater than that in the verification set out of 100,000 images, re-labeling all keypoints in the comparison set would be extremely costly in terms of both manpower and funds. However, the target detection method of this embodiment allows the comparison set to retain only bounding box labeled data.

[0074] Step S22: Use a neural network to process the feature maps of each image to obtain the image detection results of the image to be detected. The neural network is trained using a neural network training method.

[0075] After training using neural network training methods, a neural network can learn from and validate image data in a dataset that includes both bounding boxes and keypoints to determine the reliability of predicted keypoints for images without labeled keypoints. Before using a neural network for object detection, it's unnecessary to label all images with keypoints; instead, the keypoint scores can be used to estimate whether images without labeled keypoints can produce reliable predicted keypoints, rather than directly rejecting the possibility of keypoints in unlabeled images.

[0076] This invention enables keypoint learning across the entire dataset by adding a small number of keypoint annotations to an existing 2D detection and annotation dataset. This ensures improved detection performance within the current detection task framework. Customers do not need to purchase new computing hardware or camera equipment; they only need to upgrade the software on existing devices using the target detection method provided in this invention. After inputting a monocular image, accurate 2D and / or 3D image detection results can be obtained according to detection requirements, with no significant increase in computational load. It not only meets the needs of finer-grained perception algorithms in edge scene target detection tasks but also maintains compatibility with existing incremental monocular 3D detection methods using bounding boxes.

[0077] Specifically, please refer to the Multi-Task Neural Network Structure Diagram. Figure 3 The entire training process of the neural network requires inputs including a set of images to be detected and the labeled data for each image in the set. Some images in the set only have bounding boxes labeled, while others have both bounding boxes and keypoints labeled. The inputs for these different labeled images vary. Figure 3 After illustrating the neural network structure, we obtain the category, bounding box, foreground score, predicted keypoint, and keypoint score for each feature map. The neural network has three layers of feature maps, with outputs in three sizes: 12*20, 24*40, and 48*80. The first 12*20 feature map output is explained separately in detail, highlighting its components. The main difference between the three layers lies in their size, a pyramid structure designed to ensure good recall and accuracy for objects of varying sizes. If we remove the wingloss and bone loss functions for keypoint regression and the cross-entropy loss function (landmark label assigner) calculated using keypoint scores from the 12*20 feature map output, the neural network structure becomes a 2D bounding box-based object detection neural network. Finally, the predicted keypoints and keypoint scores output by the neural network are compared with the corresponding labeled data to calculate the loss function.

[0078] For details on the steps of calculating the loss function between the predicted keypoints, keypoint scores, and other numerical values ​​output by the neural network and the corresponding labeled data, please refer to the first embodiment of this invention. For example, when the neural network determines the probability that a predicted keypoint exists at the center point of the feature map, this probability is calculated using the cross-entropy loss function `cross entropy(predict, gt)`. Specifically, the neural network calculates the overall loss for each type of labeled data and keypoint score using the cross-entropy loss function `cross entropy(predict, gt)`. The calculated overall cross-entropy loss function (landmarkLabel Assigner) is expressed as follows:

[0079] Loss lmk_obj_total =Loss lmk_obj +Loss lmk_pos +Loss lmk_neg .

[0080] Overall Loss lmk_obj_total middle:

[0081] Loss lmk_obj =CrossEntropy(lmkobj pred *lmkobj gt lmkobj gt ), where lmkobj pred lmkobj is used to score key points in the output of a neural network. gt This refers to the center points of multiple key points in a manually labeled feature map. (Loss) lmk_obj The calculation of the cross-entropy loss function is used to train the neural network to learn the key points of the feature map based on the labeled data with key points, that is, to ensure that the neural network can learn how to label key points based on the labeled key points.

[0082] Loss lmk_pos =CrossEntropy(lmkobj pred *mask pos obj gt *mask pos ), where mask pos =where(obj) gt -lmkobj pred <0.005), obj gt This represents the location of the center point of the 2D bounding box (i.e., the labeled detection box) in the feature map (i.e., the first center point location mentioned below). lmk_pos The calculation of the cross-entropy loss function is used to: train the neural network to find the corresponding key points when the distance between the position of the center point of the 2D bounding box in the feature map and the key point score meets the set threshold, and determine that the center point position of the feature map of the labeled detection box has a predicted key point.

[0083] Loss lmk_neg =CrossEntropy(lmkobj pred *mask neg obj gt *mask neg ), where mask neg =where(obj) gt =0), Loss lmk_negThe cross-entropy loss function is used to train the neural network to learn feature maps (feature maps) that have neither labeled detection boxes nor labeled keypoints without requiring training. Images in these feature maps are then directly identified as background images. Therefore, this embodiment of the invention can filter out a large number of occlusion cases through keypoint scores, providing a stable image perception result for the subsequent keypoint regression decision-making process.

[0084] The steps of keypoint regression processing by a neural network based on labeled keypoints include: minimizing the distance between predicted and labeled keypoints as the first regression objective; and ensuring that the distances between predicted keypoints meet a predetermined range as the second regression objective; and determining the keypoint regression result based on the first and second regression objectives. If the first regression objective is calculated using the Wing loss function and the second regression objective is calculated using the Bone loss function, and both Wing loss and Bone loss functions are used for keypoint regression processing, the keypoint regression task can be specifically expressed by the following formula:

[0085] Loss lmk_reg =wingloss(lmk) pred ,lmk gt )+0.1*boneloss(lmk pred ,lmk gt ).

[0086] In the formula, lmk pred lmk is the location of the predicted keypoints in the neural network's prediction output. gt This represents the positions of multiple manually labeled keypoints in the image. The purpose of the Wing loss function is to minimize the distance between the predicted keypoints output by the neural network and the labeled keypoints. Specifically, the distances between predicted keypoints should meet a predetermined range. This is achieved by using the bone loss function to ensure that the distances between predicted keypoints are consistent with the pairwise distances between labeled keypoints. Furthermore, using both Wing loss and bone loss functions for keypoint regression calculations results in more accurate regression results.

[0087] This invention considers the compatibility of 2D bounding box detection and keypoint detection tasks, forming a target detection method primarily based on bounding box detection. Specifically, during neural network training, this invention allows for some data to be unannotated with keypoints; only bounding boxes need to be labeled. The neural network can then predict keypoints in each image using keypoint scores. That is, even without labeled keypoints during training, the target detection method provided by this invention can use the neural network to infer whether reliable keypoints can be output at the image center of each image based on images with keypoints, rather than directly rejecting the possibility of keypoints at the image center. Therefore, using the target detection method provided by this invention can improve the robustness of monocular 3D target detection products in general detection scenarios.

[0088] Therefore, this embodiment of the invention enables the neural network to learn the keypoint performance of all labeled data in data A and data B by simply adding some data B that simultaneously labels both detection boxes and keypoints, based on the original 2D detection labeled data A (data A is image data with only labeled detection boxes). The amount of data A is far greater than that of data B. Since relabeling all keypoints on data A would be extremely costly in terms of both manpower and funds, the object detection method of this embodiment allows data A to still be labeled data with only detection boxes, allowing the keypoints provided by data B (as a validation image set) to be used for comparative training of the neural network on data A.

[0089] Therefore, embodiments of the present invention can achieve full keypoint learning on existing 2D detection labeled datasets by adding a small amount of data with keypoint annotations, ensuring improved detection performance within the current detection task framework. This eliminates the need for customers to purchase new computing hardware and camera equipment, thereby enhancing the market competitiveness of products using the target detection method provided by the embodiments of the present invention. Furthermore, based on the foreground score obtained by the target detection neural network based on the 2D detection box, it can further filter out a large number of occlusion cases by judging the background image through keypoint scores, based on the foreground score obtained by the target detection neural network, thus providing stable perceptual results for subsequent detection outputs of the neural network.

[0090] This invention provides a unified summary and optimization for monocular 3D detection tasks, transforming them into key point detection with prior information, making them applicable in various fields. Examples include detecting 3D bounding boxes for vehicles in traffic scenarios, detecting 3D bounding boxes for surrounding vehicles in autonomous driving scenarios, detecting key points on license plates in different scenarios, and detecting 3D bounding boxes for key workpieces in robotic scenarios.

[0091] The detection method provided in this embodiment of the invention only increases the computational load by 20% on the basis of the original target detection network to complete the monocular 3D detection task, ensuring that the neural network can run smoothly on the original edge devices and end-side devices with less computation.

[0092] A third embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the neural network training method provided in the first embodiment of the present invention.

[0093] The fourth embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the neural network training method provided in the first embodiment of the present invention.

[0094] The fifth embodiment of the present invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the neural network training method provided in the first embodiment of the present invention.

[0095] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A neural network training method, characterized in that, include: Receive a set of verification images with labeled data collected from a target detection scene, and generate feature maps for each image in the set of verification images. The data types labeled by the labeled data include detection boxes and multiple key points. Calculate multiple predicted key points for each of the feature maps, and the key point score for each predicted key point; For each type of labeled data and the keypoint score, a cross-entropy loss function is calculated to determine the probability that the predicted keypoint is located at the center point of the feature map; wherein, the true value of the cross-entropy function includes simultaneously labeling the detection box and the keypoint, labeling only the detection box, and not labeling either the detection box or the keypoint; When it is determined from the probability that the predicted key point is located at the center point, the predicted key point is subjected to key point regression processing based on the marked key point to obtain the key point regression result. The key point regression result is used to indicate the offset of the predicted key point relative to the center point. The feature map is processed based on the key point regression results to obtain the image detection results for each image, and the structural parameters of the neural network are adjusted based on the image detection results and the labeled data. The loss value of the cross-entropy function is calculated in the following way: ； ,in, Scoring the key points output by the neural network. The center point location of multiple key points in the manually annotated feature map; The calculation of the cross-entropy loss function is used to train the neural network to learn the corresponding key points based on the labeled data with key points labeled, that is, to ensure that the neural network can learn how to label key points based on the labeled key points; ,in, , The position of the center point of the unannotated detection box in the feature map; The calculation of the cross-entropy loss function is used to: train the neural network to find the corresponding predicted key points when the distance between the position of the center point of the labeled detection box in the feature map and the key point score meets the set threshold, and determine that the feature map with only labeled detection boxes has key points; ,in, , The calculation of the cross-entropy loss function is used to train neural networks without learning feature maps that have neither labeled detection boxes nor labeled keypoints, and directly determines the images in the feature maps that have neither labeled detection boxes nor labeled keypoints as background images.

2. The method according to claim 1, characterized in that, Before determining the probability that the predicted keypoint exists at the center point of the feature map based on the labeled data and the keypoint score, the method further includes: Obtain the position of the first center point determined when labeling the detection box; The step of determining the probability that the predicted keypoint exists at the center point of the feature map based on the labeled data and the keypoint score includes: A loss function is calculated on the first center point location and the key point score to determine whether the predicted key point exists at the center point location based on the loss function.

3. The method according to claim 2, characterized in that, The step of calculating a loss function on the first center point location and the keypoint score, and determining whether the predicted keypoint is present at the center point location based on the loss function, includes: With minimizing the loss function as the calculation objective, when the difference between the first center point location and the key point score is less than a preset threshold, it is determined that the predicted key point exists at the center point location.

4. The method according to claim 1, characterized in that, The steps for performing keypoint regression processing on the predicted keypoints based on the labeled keypoints include: The regression result of the key points is determined by minimizing the distance between the predicted key points and the labeled key points.

5. The method according to claim 1, characterized in that, The step of performing keypoint regression processing on the predicted keypoints based on the labeled keypoints further includes: The first regression objective is to minimize the distance between the predicted keypoints and the labeled keypoints. The second regression objective is to ensure that the distance between the predicted keypoints meets a predetermined distance range. The regression result of the keypoints is determined based on the first regression objective and the second regression objective.

6. The method according to claim 1, characterized in that, Before determining the probability that the predicted keypoint exists at the center point of the feature map based on the labeled data and the keypoint score, the method further includes: Based on a preset score threshold range, the scores of the key points within the score threshold range are filtered out, so as to determine the probability that the predicted key point is located at the center point of the feature map based on the labeled data and the filtered key point scores.

7. The method according to claim 1, characterized in that, Before determining the probability that the predicted keypoint exists at the center point of the feature map based on the labeled data and the keypoint score, the method further includes: Obtain the prediction bounding box determined based on the predicted key points, and determine the degree of overlap between the prediction bounding box and the labeled detection bounding box. Based on the degree of overlap, filter out the images with positive sample detection bounding boxes.

8. A target detection method, characterized in that, The method includes: Receive a set of images to be detected with labeled data collected from a target detection scene, and generate feature maps of each image to be detected in the set of images to be detected, wherein the data type labeled by the labeled data includes at least detection boxes; The feature maps of each image are processed using a neural network to obtain the image detection result of the image to be detected, wherein the neural network is trained by any one of the neural network training methods described in claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images