Method, device and autonomous vehicle for generating an object detection model

Abstract

This application discloses a method, apparatus, and autonomous vehicle for generating an object detection model, relating to the field of autonomous driving. One specific implementation of the method includes: acquiring a target sample image; inputting the target sample image into a backbone network included in a machine learning model to obtain a feature map of the target sample image; generating a heatmap based on the target sample image and the feature map; classifying each location point within the heatmap using a classification network included in the machine learning model, and determining a classification loss value based on the classification results corresponding to each location point and the category labels of each location point within the heatmap; performing regression processing on each location point within the heatmap using a regression network included in the machine learning model, and determining a regression loss value based on the regression results corresponding to each location point and the regression labels of each location point within the heatmap; and training the machine learning model based on the classification loss value and the regression loss value to obtain an object detection model.

Description

Technical Field

[0001] This application relates to the field of computer vision, specifically to the field of autonomous driving, and particularly to methods, apparatus, and autonomous vehicles for generating object detection models. Background Technology

[0002] In the field of autonomous driving, object detection involves identifying foreground objects in visual images and determining their category, location, and pose. Compared to LiDAR and ultrasonic recognition technologies, it offers lower sensor costs for environmental perception.

[0003] Currently, object detection detectors in related technologies use Gaussian Focal Loss to calculate the classification loss, while the regression loss is calculated using the bounding boxes of the samples. Gaussian Focal Loss is a loss function calculation method used by keypoint detection detectors like CenterNet / ConerNet. This Gaussian Focal Loss is a loss function calculation method for discrete sample labels. When calculating the classification loss function from the heatmap generated by keypoints, this method sets Gaussian vertices as positive samples and the rest as negative samples; the regression loss only selects center points as positive samples. This method of using hard thresholds to set positive and negative samples for continuous classification labels makes it difficult for the model to obtain the optimal solution. Summary of the Invention

[0004] Embodiments of this application propose a method, apparatus, and autonomous vehicle for generating object detection models.

[0005] In a first aspect, embodiments of this application provide a method for generating an object detection model. The method includes: acquiring a target sample image; inputting the target sample image into a backbone network included in a machine learning model to obtain a feature map of the target sample image; processing the target sample image and the feature map to generate a heatmap; classifying each location point within the heatmap using a classification network included in the machine learning model, and determining a classification loss value based on the classification results corresponding to each location point and the category labels of each location point within the heatmap; performing regression processing on each location point within the heatmap using a regression network included in the machine learning model, and determining a regression loss value based on the regression results corresponding to each location point and the regression labels of each location point within the heatmap; and training the machine learning model based on the classification loss value and the regression loss value to obtain an object detection model.

[0006] In some embodiments, processing the target sample image and feature map to generate a heatmap includes: projecting the bounding box containing the target object in the target sample image onto the feature map to obtain a two-dimensional image; determining the radius of a Gaussian circle based on the target location information; and determining the heatmap based on the radius of the Gaussian circle.

[0007] In some embodiments, the target location information includes target size information; determining a heatmap based on the radius of a Gaussian circle includes: determining the radius of a Gaussian circle based on a preset Gaussian function and a preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box containing the target object in the target sample image and the bounding box containing the target object in the two-dimensional image; and determining a heatmap based on the radius of the Gaussian circle and the target size information.

[0008] In some embodiments, the target object in the target sample image is located on the boundary of the feature map, and the target location information includes target coordinate information and target size information; the method for generating the target detection model further includes: determining the distance information between the center point of the bounding box where the target object is located in the two-dimensional image and the bounding box where the target object is located in the two-dimensional image based on the target coordinate information; and determining the corrected size information based on the distance information and the target size information.

[0009] In some embodiments, determining a heatmap based on the radius of a Gaussian circle includes: determining the radius of a Gaussian circle based on a preset Gaussian function and a preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box containing the target object in the target sample image and the bounding box containing the target object in the two-dimensional image; and determining a heatmap based on the radius of the Gaussian circle and the corrected size information.

[0010] In some embodiments, each location point within the heatmap includes: a location point corresponding to each target sample image within the radius of a Gaussian circle centered on the coordinate information of the center point of the target object.

[0011] In some embodiments, training a machine learning model based on classification loss and regression loss to obtain an object detection model includes: normalizing the regression loss and classification loss to obtain the regression loss and classification loss corresponding to each location point; and training the machine learning model based on the classification loss and corresponding first preset weight, and the regression loss and corresponding second preset weight for each location point to obtain an object detection model.

[0012] Secondly, embodiments of this application provide an apparatus for generating an object detection model. The apparatus includes: an image acquisition module configured to acquire a target sample image; a feature map acquisition module configured to input the target sample image into a backbone network included in a machine learning model to obtain a feature map of the target sample image; a heatmap generation module configured to process the target sample image and the feature map to generate a heatmap; a first determination module configured to classify each location point within the heatmap using a classification network included in the machine learning model, and determine a classification loss value based on the classification results corresponding to each location point and the category labels of each location point within the heatmap; a second determination module configured to perform regression processing on each location point within the heatmap using a regression network included in the machine learning model, and determine a regression loss value based on the regression results corresponding to each location point and the regression labels of each location point within the heatmap; and a model acquisition module configured to train the machine learning model based on the classification loss value and the regression loss value to obtain an object detection model.

[0013] In some embodiments, the heatmap generation module includes: an image projection unit configured to project the bounding box containing the target object in the target sample image onto a feature map to obtain a two-dimensional image; a radius determination unit configured to determine the radius of a Gaussian circle based on the target location information; and a heatmap determination unit configured to determine a heatmap based on the radius of the Gaussian circle.

[0014] In some embodiments, the target location information includes target size information; the heatmap determination unit is further configured to: determine the radius of a Gaussian circle based on a preset Gaussian function and a preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box of the target object in the target sample image and the bounding box of the target object in the two-dimensional image; and determine a heatmap based on the radius of the Gaussian circle and the target size information.

[0015] In some embodiments, the target object in the target sample image is located on the boundary of the feature map, and the target position information includes target coordinate information and target size information; the apparatus for generating the target detection model further includes: a distance determination module, configured to determine the distance information between the center point of the bounding box where the target object is located in the two-dimensional image and the bounding box where the target object is located in the two-dimensional image based on the target coordinate information; and a size determination module, configured to determine the corrected size information based on the distance information and the target size information.

[0016] In some embodiments, the heatmap determination unit is further configured to: determine the radius of a Gaussian circle based on a preset Gaussian function and a preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box containing the target object in the target sample image and the bounding box containing the target object in the two-dimensional image; and determine the heatmap based on the radius of the Gaussian circle and the corrected size information.

[0017] In some embodiments, each location point within the heatmap includes: a location point corresponding to each target sample image within the radius of a Gaussian circle centered on the coordinate information of the center point of the target object.

[0018] In some embodiments, the model obtaining module is further configured to: normalize the regression loss value and the classification loss value to obtain the regression loss value and the classification loss value corresponding to each location point; train the machine learning model based on the classification loss value and the corresponding first preset weight, and the regression loss value and the corresponding second preset weight corresponding to each location point to obtain the object detection model.

[0019] Thirdly, embodiments of this application provide a target detection method, the method comprising: acquiring a test image; inputting the test image into a target detection model as described in the first or second aspect to obtain an output result; and determining the category and location of a target object in the test image based on the output result.

[0020] In some embodiments, determining the category and location of a target object in the image under test based on the output results includes: determining the category of the target object in the image under test based on the classification processing results included in the output results; and determining the location of the target object in the image under test based on the regression processing results included in the output results.

[0021] Fourthly, embodiments of this application provide a target detection apparatus, the apparatus comprising: an image acquisition module configured to acquire an image to be tested; an image input module configured to input the image to be tested into a target detection model as described in the first or second aspect; and a result determination module configured to determine the category and location of a target object in the image to be tested based on the output result of the target detection model.

[0022] In some embodiments, the result determination module is further configured to: determine the category of the target object in the image to be tested based on the classification processing result output by the target detection model; and determine the location of the target object in the image to be tested based on the regression processing result output by the target detection model.

[0023] Fifthly, embodiments of this application provide an electronic device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described in the first or third aspect.

[0024] In a sixth aspect, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to cause a computer to perform the methods described in the first or third aspect.

[0025] In a seventh aspect, embodiments of this application provide an autonomous vehicle, the autonomous vehicle comprising: the apparatus for generating a target detection model as described in the second aspect and / or the target detection apparatus as described in the fourth aspect.

[0026] The method, apparatus, and autonomous vehicle for generating object detection models provided in the embodiments of this application first acquire a target sample image; then, the target sample image is input into the backbone network included in the machine learning model to obtain a feature map of the target sample image; then, the target sample image and the feature map are processed to generate a heatmap; then, the classification network included in the machine learning model is used to classify each location point in the heatmap, and a classification loss value is determined based on the classification processing result corresponding to each location point and the category label of each location point in the heatmap; then, the regression network included in the machine learning model is used to regress each location point in the heatmap, and a regression loss value is determined based on the regression processing result corresponding to each location point and the regression label of each location point in the heatmap; finally, the machine learning model is trained based on the classification loss value and the regression loss value to obtain an object detection model. This method can obtain continuous classification labels to determine the corresponding classification loss value and continuous regression labels to determine the corresponding regression loss value for each location in the heatmap generated by processing the target sample image and the feature map, achieving global optimization; furthermore, it can solve the model training instability caused by the discrete label loss calculation method under hard thresholding, thus significantly improving the performance of the object detection model. Attached Figure Description

[0027] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of an embodiment of the method for generating an object detection model according to this application; Figure 3 This is a schematic diagram of a heatmap generated based on the projection of the target sample image onto the feature map; Figure 4 This is a flowchart of an embodiment of the method for generating an object detection model according to this application; Figure 5 This is a flowchart of an embodiment of the target detection method according to this application; Figure 6This is a schematic diagram of an embodiment of the apparatus for generating a target detection model according to this application; Figure 7 This is a schematic diagram of the structure of an embodiment of the target detection device according to this application; Figure 8 This is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of this application; Figure 9 This is a structural schematic diagram of an embodiment of an autonomous vehicle according to this application; Figure 10 This is a structural schematic diagram of an embodiment of an autonomous vehicle according to this application. Detailed Implementation

[0028] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Figure 1 An exemplary system architecture 100 is shown, which can be applied to the method and apparatus for generating target detection models or embodiments of the target detection method and apparatus of this application.

[0031] like Figure 1 As shown, system architecture 100 may include terminal devices 101 and 102, network 103, and server 104. Network 103 serves as the medium for providing communication links between terminal devices 101, 102, and server 104. Network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0032] Users can use terminal devices 101 and 102 to interact with server 104 via network 103, for example, by transmitting target sample images. Various client applications and intelligent interactive applications, such as image processing applications and video processing applications, can be installed on terminal devices 101 and 102.

[0033] Terminal devices 101 and 102 can be either hardware or software. When terminal devices 101 and 102 are hardware, they can be electronic products that allow human-computer interaction with users through one or more methods such as a keyboard, touchpad, touchscreen, remote control, voice interaction, or handwriting devices. Examples include PCs (Personal Computers), mobile phones, smartphones, PDAs (Personal Digital Assistants), wearable devices, PPCs (Pocket PCs), tablets, smart car systems, smart TVs, smart speakers, laptops, and desktop computers. When terminal devices 101 and 102 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0034] Server 104 can provide various services. For example, server 104 can acquire target sample images; input the target sample images into the backbone network included in the machine learning model to obtain feature maps of the target sample images; process the target sample images and feature maps to generate heatmaps; classify each location point in the heatmap using the classification network included in the machine learning model, and determine the classification loss value based on the classification results corresponding to each location point and the category labels of each location point in the heatmap; perform regression processing on each location point in the heatmap using the regression network included in the machine learning model, and determine the regression loss value based on the regression results corresponding to each location point and the regression labels of each location point in the heatmap; train the machine learning model based on the classification loss value and the regression loss value to obtain an object detection model.

[0035] It should be noted that server 104 can be either hardware or software. When server 104 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 104 is software, it can be implemented as multiple software programs or software modules (for example, used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0036] It should be noted that the method for generating a target detection model or the target detection method provided in the embodiments of this application are generally executed by the server 104, and correspondingly, the device for generating a target detection model or the target detection device is generally located in the server 104.

[0037] It should be understood that Figure 1 The number of electronic devices, networks, and servers shown is merely illustrative. Any number of electronic devices, networks, and servers can be included depending on implementation needs.

[0038] like Figure 2 The flowchart 200 illustrates an embodiment of a method for generating an object detection model according to this application, which may include the following steps: Step 201: Obtain the target sample image.

[0039] In this embodiment, the execution entity of the method for generating the target detection model (e.g.) Figure 1 The terminal devices 101 and 102 (or server 104) shown can acquire the target sample image. The target sample image can be acquired by the terminal device (e.g.,...). Figure 1 The terminal devices 101 and 102 shown) or the server (e.g.) Figure 1 The image captured by the imaging device of the server 104 shown can be a target sample image, which can be multiple images or multiple frames from a video. The target sample image includes one or more target objects. It can be used to train a target detection model to identify any one of the following: category, location, or pose of the target object within the target sample image. The target object can refer to a general type of object, such as a face, vehicle, or pedestrian. In this case, target object detection can be face detection, vehicle detection, or pedestrian detection, with the aim of detecting faces, vehicles, or pedestrians in the image. The target object can also be an object with specific characteristics that distinguish it from other objects of the same type, such as the face of a specific user. In this case, the purpose of target object detection can be to detect the face of the specific user to determine whether the specific user's face exists in the target sample image, and, if the specific user's face exists in the target sample image, to determine the location of the specific user's face.

[0040] Step 202: Input the target sample image into the backbone network of the machine learning model to obtain the feature map of the target sample image.

[0041] In this embodiment, the aforementioned execution entity can input the target sample image into the backbone network included in the machine learning model to obtain the feature map of the target sample image. The backbone network is a network used for feature extraction. This backbone network can include multiple convolutional layers or be implemented using various network structures. It can be used to extract features from the input target sample image and generate the feature map of the target sample image. The feature map can be an image generated after extracting key information from the target sample image. The aforementioned machine learning model can be a probabilistic model, classification model, or other classifier from existing or future technologies. For example, the machine learning model can include any of the following: Faster R (Region)-CNN (Convolutional Neural Networks), SSD (SingleShot MultiBox Detector), YOLD (You only look once), R-CNN, Fast R-CNN, or R-FCN (Object Detection via Region-based Fully Convolutional Networks).

[0042] In one example, the input target sample image has a width of W and a height of H. This target sample image is processed by a backbone network to extract a feature map, and the dimension of this feature map is W / 4. H / 4 C represents the number of channels, indicating that the generated feature map has been downsampled by a factor of 4.

[0043] In one example, the backbone network described above may include a residual network and a feature pyramid network.

[0044] Among them, residual networks are a type of deep convolutional network. For example, residual networks can be implemented using ResNet (Residual Neural Network).

[0045] Feature Pyramid Networks (FPNs) can be expanded into a standard convolutional network through top-down channels and lateral connections, thereby effectively extracting rich, multi-scale feature pyramid images from single-resolution input images. FPNs contain a multi-layered structure, with each layer capable of detecting the image at different scales, generating multi-scale feature maps. FPNs can significantly improve the multi-scale prediction capabilities of Fully Convolutional Networks (FCNs).

[0046] Correspondingly, in this example, the Feature Pyramid Network can have multiple FPN layers. In practical applications, the number of FPN layers in the Feature Pyramid Network can be flexibly increased or decreased according to actual needs.

[0047] It should be noted that the more layers an FPN has, the larger the range of its receptive field, the better it can detect objects of different scales, and the stronger its size invariance. The receptive field can be defined as the region of the target sample image that the convolutional neural network's features can see.

[0048] Step 203: Process the target sample image and feature map to generate a heatmap.

[0049] In this embodiment, the aforementioned execution entity can generate a heatmap of the target sample image based on the processing results of the target sample image and the feature map. Optionally, the feature map has a dimension of W / 4. H / 4 C represents the number of channels in the generated feature map, which has undergone a 4x downsampling process. At this point, the heatmap (e.g., ...) Figure 3 The resolution corresponding to the "white spot" in the image is W / 4. H / 4.

[0050] In one example, processing the target sample image and feature map to generate a heatmap may include: projecting the target sample image onto the feature map, and then convolving the projected image to obtain the heatmap. The heatmap may include at least one thermal distribution region, which can be presented as a circular region radiating outwards from the center. The thermal value is highest at the center of this region, and decreases further away from the center. If the heatmap is used to represent the thermal distribution of the center point of the bottom surface of a 3D bounding box, the position of the center point in the target sample image can be determined based on the center point of the thermal distribution region when making predictions using an object detection model.

[0051] Step 204: Classify each location point in the heatmap using the classification network included in the machine learning model, and determine the classification loss value based on the classification results corresponding to each location point and the category labels of each location point in the heatmap.

[0052] In this embodiment, the aforementioned execution entity can classify each location point within the heatmap using a classification network included in the machine learning model, obtaining the classification result corresponding to each location point. Then, based on the classification result corresponding to each location point and the category label of each location point in the heatmap, the classification loss value corresponding to each location point is determined. The aforementioned classification result can be the result obtained by the classification network classifying each location point within the heatmap. The aforementioned category label can refer to the category label data of the target object in the heatmap.

[0053] The classification subnet can include multiple convolutional layers, which can be used to classify target objects.

[0054] In one example, a classification network can be used to determine whether there are objects belonging to the category of interest (i.e., the target objects in the target sample image) in a target sample image, and output the probability that objects belonging to the corresponding category of interest appear in the image.

[0055] The loss value determines how close the actual output is to the expected output. A smaller loss value indicates that the actual output is closer to the expected output. This can be understood as the classification result being the actual output of the classification network, while the class label is the expected output. By calculating the difference between the classification result and the class label, we can determine how close the actual output is to the expected output.

[0056] In one example, determining the classification loss value may include:

[0057] Among them, L cls For classification loss, y is the label value of each location point in the heatmap in step 203 (i.e., the classification label of each location point in the heatmap), σ is the category score of the simoid output of the classification branch (i.e., the classification result), and β is a hyperparameter, which is set to 2 by default.

[0058] During the training of the object detection model, the loss function is determined by the generated heatmap; the backpropagation algorithm updates the parameters of the object detection model more effectively to achieve global optimum, thereby solving the model training instability caused by the discrete label loss calculation method under hard threshold in related technologies, and significantly improving the performance of the object detection model.

[0059] Step 205: Perform regression processing on each location point in the heatmap using the regression network included in the machine learning model, and determine the regression loss value based on the regression processing results corresponding to each location point and the regression labels of each location point in the heatmap.

[0060] In this embodiment, the aforementioned execution entity can perform regression processing on each location point within the heatmap using a regression network included in the machine learning model, obtaining the regression processing result corresponding to each location point; then, based on the regression processing result corresponding to each location point and the regression label of each location point within the heatmap, a regression loss value is determined. The aforementioned regression processing result can be the result obtained by the regression network performing regression processing on the heatmap. The aforementioned regression label can be the location label data of the target object in the heatmap.

[0061] The regression subnet can include multiple convolutional layers; this regression subnet can be used to locate target objects.

[0062] In one example, a regression network can be used to determine the location of a target object of the class of interest in a target sample image, typically outputting the bounding box of the target object. For instance, in a target object detection task, the regression network can output the coordinates of the regression box of the target object. The regression box can be the bounding box of the target object predicted by the regression network, which can be used to characterize the specific location of the target object.

[0063] The regression result can be the actual output of the regression network, while the regression label is the expected output of the regression network. By calculating the regression result and the category label, we can know how close the actual output of the regression network is to the expected output.

[0064] It should be noted that steps 204 and 205 can be executed simultaneously; or step 205 can be executed first, followed by step 204; or step 204 can be executed first, followed by step 205. These will not be elaborated further here.

[0065] Step 206: Train the machine learning model based on the classification loss value and the regression loss value to obtain the object detection model.

[0066] In this embodiment, the aforementioned execution entity can train the machine learning model based on the classification loss value and the regression loss value corresponding to each location point to obtain the object detection model.

[0067] In one example, a backpropagation algorithm can be used to jointly train the backbone network, classification network, and regression network based on classification and regression loss values. Training stops when the classification loss value converges to a first preset value and the regression loss value converges to a second preset value. That is, the network parameters can be adjusted backward based on the loss values ​​until the loss values ​​reach the iteration cutoff condition, at which point training ends, confirming that the network meets the requirements and can output the expected results, thus achieving object detection. The aforementioned iteration cutoff condition can be set by the user or based on the recognition accuracy of the object detection model.

[0068] The method for generating an object detection model provided in this embodiment first acquires a target sample image; then, the target sample image is input into the backbone network of the machine learning model to obtain a feature map of the target sample image; then, the target sample image and feature map are processed to generate a heatmap; then, the classification network of the machine learning model classifies each location point in the heatmap, and determines the classification loss value based on the classification results corresponding to each location point and the class labels of each location point in the heatmap; then, the regression network of the machine learning model performs regression processing on each location point in the heatmap, and determines the regression loss value based on the regression results corresponding to each location point and the regression labels of each location point in the heatmap; finally, the machine learning model is trained based on the classification loss value and the regression loss value to obtain the object detection model. This method can obtain continuous classification labels to determine the corresponding classification loss value and continuous regression labels to determine the corresponding regression loss value for each location in the heatmap generated by processing the target sample image and feature map, achieving global optimization; furthermore, it can solve the model training instability caused by the discrete label loss calculation method under hard thresholding, thus significantly improving the performance of the object detection model.

[0069] In some optional implementations of this embodiment, each location point within the heatmap includes: the location point corresponding to each target sample image in all target sample images within the radius of a Gaussian circle centered on the coordinate information of the center point of the target object.

[0070] In some optional implementations of this embodiment, training the machine learning model based on classification loss and regression loss to obtain an object detection model includes: normalizing the regression loss to obtain the regression loss corresponding to each location point; and training the machine learning model based on the classification loss and the corresponding first preset weight, as well as the regression loss corresponding to each location point and the corresponding second preset weight, to obtain the object detection model.

[0071] In one example, the radius (r_w, r_h) of the Gaussian circle of the target object on the feature map can be obtained from the above, and the projection position of the center point of the target object on the feature map is: ,by N samples within the radius of the central Gaussian circle participate in the calculation of regression and classification loss values. The regression and classification loss values ​​are normalized according to the number of samples to obtain the regression loss value Lreg and the classification loss value for each location point. : L reg =λ1 / N(B1 -B1) L cls =λ2 / N(B2 -B2) Where λ1 is the weight of the regression loss corresponding to each location point (i.e., the second preset weight), B1 B1 is the predicted value (i.e., the regression result), B2 is the true value (i.e., the regression label); λ2 is the weight of the classification loss corresponding to each location point (i.e., the first preset weight), and B3 is the true value (i.e., the regression label). B1 is the predicted value (i.e., the classification result), and B2 is the true value (i.e., the classification label).

[0072] It should be noted that the first preset weight and the second preset weight can be set by the user based on experience or based on the recognition accuracy of the target detection model.

[0073] In this implementation, multiple samples within the radius of the Gaussian circle in the heatmap are used to participate in the regression loss calculation, which avoids the problem of insufficient sample quantity and unstable convergence caused by a single central sample point, and achieves better detection performance.

[0074] like Figure 4 The flowchart 400 illustrates an embodiment of a method for generating an object detection model according to this application, which may include the following steps: Step 401: Obtain the target sample image.

[0075] In this embodiment, the execution entity of the method for generating the target detection model (e.g.) Figure 1 The terminal devices 101 and 102 (or server 104) shown can acquire the target sample image. The target sample image can be acquired by the terminal device (e.g.,...). Figure 1 The terminal devices 101 and 102 shown) or the server (e.g.) Figure 1 The image captured by the imaging device of the server 104 shown can be a target sample image, which can be multiple images or multiple frames from a video. The target sample image includes one or more target objects. It can be used to train a target detection model to identify any one of the following: category, location, or pose of the target object within the target sample image. The target object can refer to a general type of object, such as a face, vehicle, or pedestrian. In this case, target object detection can be face detection, vehicle detection, or pedestrian detection, with the aim of detecting faces, vehicles, or pedestrians in the image. The target object can also be an object with specific characteristics that distinguish it from other objects of the same type, such as the face of a specific user. In this case, the purpose of target object detection can be to detect the face of the specific user to determine whether the specific user's face exists in the target sample image, and, if the specific user's face exists in the target sample image, to determine the location of the specific user's face.

[0076] Step 402: Input the target sample image into the backbone network of the machine learning model to obtain the feature map of the target sample image.

[0077] In this embodiment, the aforementioned execution entity can input the target sample image into the backbone network included in the machine learning model to obtain the feature map of the target sample image. The backbone network is a network used for feature extraction. This backbone network can include multiple convolutional layers or be implemented using various network structures. It can be used to extract features from the input target sample image and generate the feature map of the target sample image. The feature map can be an image generated after extracting key information from the target sample image. The aforementioned machine learning model can be a probabilistic model, classification model, or other classifier from existing or future technologies. For example, the machine learning model can include any of the following: Faster R (Region)-CNN (Convolutional Neural Networks), SSD (SingleShot MultiBox Detector), YOLD (You only look once), R-CNN, Fast R-CNN, or R-FCN (Object Detection via Region-based Fully Convolutional Networks).

[0078] In one example, the backbone network described above may include a residual network and a feature pyramid network.

[0079] Among them, residual networks are a type of deep convolutional network. For example, residual networks can be implemented using ResNet (Residual Neural Network).

[0080] Feature Pyramid Networks (FPNs) can be expanded into a standard convolutional network through top-down channels and lateral connections, thereby effectively extracting rich, multi-scale feature pyramid images from single-resolution input images. FPNs contain a multi-layered structure, with each layer capable of detecting the image at different scales, generating multi-scale feature maps. FPNs can significantly improve the multi-scale prediction capabilities of Fully Convolutional Networks (FCNs).

[0081] Correspondingly, in this example, the Feature Pyramid Network can have multiple FPN layers. In practical applications, the number of FPN layers in the Feature Pyramid Network can be flexibly increased or decreased according to actual needs.

[0082] It should be noted that the more layers an FPN has, the larger the range of its receptive field, the better it can detect objects of different scales, and the stronger its size invariance. The receptive field can be defined as the region of the target sample image that the convolutional neural network's features can see.

[0083] Step 403: Project the bounding box containing the target object in the target sample image onto the feature map to obtain a two-dimensional image.

[0084] In this implementation, the execution entity can project the bounding box containing the target object in the target sample image onto the feature map to obtain a two-dimensional image. The target sample image can be a three-dimensional image.

[0085] Here, the bounding box containing the target object in the target sample image can be the bounding box surrounding the target object in the target sample image. Optionally, the execution entity can manually calibrate the target sample image to obtain the bounding box of the target object in the target sample image; or, the execution entity can also use a polygon approximation detection method to obtain the bounding box of the target object in the target sample image.

[0086] In one example, the aforementioned execution entity can project the bounding box of the target object in the target sample image onto the feature map based on coordinate transformation, and determine the position and / or type of the target object based on the projected two-dimensional image. Since the possible position of the target object (i.e., the area enclosed by the aforementioned bounding box) is determined in advance based on the target sample image, the detection area is reduced by projecting the bounding box of the target object in the target sample image onto the feature map. Target detection only needs to be performed on the small area corresponding to the bounding box in the feature map, instead of performing target detection on the entire feature map, which greatly reduces the computational load, improves the detection speed, and ensures detection accuracy.

[0087] Optionally, the execution entity can also determine a coordinate transformation matrix based on the calibration parameters of the camera that captured the target sample image; then, based on this coordinate transformation matrix, the bounding box of the target object in the target sample image can be projected onto the feature map to obtain a two-dimensional image. The aforementioned coordinate transformation matrix can be the matrix used for coordinate transformation during projection.

[0088] Correspondingly, in this example, the bounding box of the target object in the three-dimensional target sample image is projected onto the feature map to obtain a two-dimensional image. The target position information of the bounding box of the target object in the two-dimensional image is (x, y, w, h), where x can be the x-coordinate of the center point, y can be the y-coordinate of the center point, w can be the width of the bounding box of the target object in the two-dimensional image, and h can be the height of the bounding box of the target object in the two-dimensional image.

[0089] Step 404: Determine the radius of the Gaussian circle based on the target location information.

[0090] In this implementation, the execution entity can determine the radius of the Gaussian circle based on the target location information. This target location information can then be used to label the position of the target object in a two-dimensional image.

[0091] In one example, the radius of the Gaussian circle can be determined based on a preset Gaussian function and the target location information. Alternatively, a preset mapping software can be used to generate the radius of the Gaussian circle based on the target location information.

[0092] Step 405: Determine the heat map based on the radius of the Gaussian circle.

[0093] In this implementation, the aforementioned execution entity can fit the radius of the Gaussian circle to obtain a heatmap. Alternatively, it can use image processing software to plot the radius of the Gaussian circle to obtain a heatmap.

[0094] Step 406: Classify each location point in the heatmap using the classification network included in the machine learning model, and determine the classification loss value based on the classification results corresponding to each location point and the category labels of each location point in the heatmap.

[0095] In this embodiment, the aforementioned execution entity can classify each location point within the heatmap using a classification network included in the machine learning model, obtaining the classification result corresponding to each location point. Then, based on the classification result corresponding to each location point and the category label of each location point in the heatmap, the classification loss value corresponding to each location point is determined. The aforementioned classification result can be the result obtained by the classification network classifying each location point within the heatmap. The aforementioned category label can refer to the category label data of the target object in the heatmap.

[0096] The classification subnet can include multiple convolutional layers, which can be used to classify target objects.

[0097] In one example, a classification network can be used to determine whether there are objects belonging to the category of interest (i.e., the target objects in the target sample image) in a target sample image, and output the probability that objects belonging to the corresponding category of interest appear in the image.

[0098] The loss value determines how close the actual output is to the expected output. A smaller loss value indicates that the actual output is closer to the expected output. This can be understood as the classification result being the actual output of the classification network, while the class label is the expected output. By calculating the difference between the classification result and the class label, we can determine how close the actual output is to the expected output.

[0099] Step 407: Perform regression processing on each location point in the heatmap using the regression network included in the machine learning model, and determine the regression loss value based on the regression processing results corresponding to each location point and the regression labels of each location point in the heatmap.

[0100] In this embodiment, the aforementioned execution entity can perform regression processing on each location point within the heatmap using a regression network included in the machine learning model, obtaining the regression processing result corresponding to each location point; then, based on the regression processing result corresponding to each location point and the regression label of each location point within the heatmap, a regression loss value is determined. The aforementioned regression processing result can be the result obtained by the regression network performing regression processing on the heatmap. The aforementioned regression label can be the location label data of the target object in the heatmap.

[0101] The regression subnet can include multiple convolutional layers; this regression subnet can be used to locate target objects.

[0102] In one example, a regression network can be used to determine the location of a target object of the class of interest in a target sample image, typically outputting the bounding box of the target object. For instance, in a target object detection task, the regression network can output the coordinates of the regression box of the target object. The regression box can be the bounding box of the target object predicted by the regression network, which can be used to characterize the specific location of the target object.

[0103] The regression result can be the actual output of the regression network, while the regression label is the expected output of the regression network. By calculating the regression result and the category label, we can know how close the actual output of the regression network is to the expected output.

[0104] It should be noted that steps 406 and 407 can be executed simultaneously; or step 407 can be executed first, followed by step 406; or step 406 can be executed first, followed by step 407. These will not be elaborated further here.

[0105] Step 408: Train the machine learning model based on the classification loss value and the regression loss value to obtain the object detection model.

[0106] In this embodiment, the aforementioned execution entity can train the machine learning model based on the classification loss value and the regression loss value corresponding to each location point to obtain the object detection model.

[0107] In one example, a backpropagation algorithm can be used to jointly train the backbone network, classification network, and regression network based on classification and regression loss values. Training stops when the classification loss value converges to a first preset value and the regression loss value converges to a second preset value. That is, the network parameters can be adjusted backward based on the loss values ​​until the loss values ​​reach the iteration cutoff condition, at which point training ends, confirming that the network meets the requirements and can output the expected results, thus achieving object detection. The aforementioned iteration cutoff condition can be set by the user or based on the recognition accuracy of the object detection model.

[0108] In this embodiment, the specific operations of steps 401, 402, 406, 407, and 408 are the same as... Figure 2 The operations of steps 201, 202, 204, 205 and 206 in the illustrated embodiments are basically the same, and will not be described again here.

[0109] from Figure 4 It can be seen from this that, with Figure 2 Compared to the corresponding embodiments, the method 400 for generating the target detection model in this embodiment first projects the bounding box containing the target object in the target sample image onto the feature map to obtain a two-dimensional image; then, based on the target position information, it determines the radius of the Gaussian circle; and then, based on the radius of the Gaussian circle, it determines the heatmap. It is possible to determine the radius of the Gaussian circle based on the target position information of the bounding box containing the target object in the two-dimensional image obtained by projecting the target sample image onto the feature map, thereby enabling the determination of the heatmap based on the radius of the Gaussian circle.

[0110] In some optional implementations of this embodiment, the target location information includes target size information; determining the heatmap based on the radius of the Gaussian circle may include: determining the radius of the Gaussian circle based on a preset Gaussian function and a preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box of the target object in the target sample image and the bounding box of the target object in the two-dimensional image; and determining the heatmap based on the radius of the Gaussian circle and the target size information.

[0111] In one example, the radius of the Gaussian circle described above can be determined based on the following steps: Assuming the preset overlap threshold is 0.5, it should be noted that the overlap threshold can be set based on the user's experience or the recognition accuracy of the target detection model.

[0112] (1) Calculate a, b, and c respectively according to the following formulas:

[0113] Then calculate r1 using the following formula:

[0114] (2) Calculate a, b, and c respectively according to the following formulas:

[0115] Then calculate r2 using the following formula:

[0116] (3) Calculate a, b, and c respectively according to the following formulas.

[0117] Then calculate r3 according to the following formula:

[0118] Calculate the radius r of the Gaussian circle based on r1, r2, and r3: r = min(r1, r2, r3) It should be noted that a, b, and c above are all intermediate variables involved in the calculation of the radius of the Gaussian circle.

[0119] In one example, generating a heatmap corresponding to the radius of a Gaussian circle could include: Based on the radius (r) of the Gaussian circle of the target object on the feature map obtained above, the diameter of the circle is calculated to be d=2. r. Assume the standard deviation of the Gaussian function of the circle is sigma = d / 6. The heatmap h(x, y) of the target object can be calculated based on the preset Gaussian function:

[0120] It should be noted that sigma represents the standard deviation, which is used to smooth images and reduce image noise.

[0121] In this implementation, additional image smoothing processing can be added, or the image can be left unprocessed; both options are valid and fall within the scope of protection of this application.

[0122] Without smoothing the image, the heatmap h(x, y) of the target object can be:

[0123] In some optional implementations of this embodiment, after projecting the bounding box containing the target object in the target sample image onto the feature map, if the target object in the target sample image is located on the boundary of the feature map, and the target position information includes target coordinate information and target size information, the method for generating the target detection model may further include: Based on the target coordinate information, determine the distance between the center point of the bounding box containing the target object in the 2D image and the bounding box containing the target object in the 2D image; based on the distance information and the size information, determine the corrected size information.

[0124] In this implementation, the execution entity can determine the distance between the center point of the border containing the target object in the two-dimensional image and the border containing the target object in the two-dimensional image based on the target coordinate information; then, it can determine the corrected size information based on the distance information and the target size information.

[0125] In one example, when the target object is at the feature map boundary, the 2D outermost bounding box of the 3D projection bounding box on the image exceeds the image boundary; this 2D outermost bounding box is truncated by the feature map boundary. To accurately calculate the heatmap of the target object on the feature map at this location, the width and height of this 2D outermost bounding box need to be corrected. Based on the target position information (x, y, w, h) of the bounding box containing the target object in the 2D image, the distance information from the center point to the bounding box containing the target object in the 2D image can be calculated as l, r, t, d. The corrected size information (w1, h1) is: w1=2 max(l, r) h1=2 max(t, d).

[0126] Where w1 is the height of the border in the corrected 2D image, h1 is the height of the border in the corrected 2D image, w is the height of the border in the 2D image, h is the height of the border in the 2D image, x is the x-coordinate of the center point in the 2D image, y is the y-coordinate of the center point in the 2D image; l is the distance from the center point in the 2D image to the top border of the 2D image; r is the distance from the center point in the 2D image to the bottom border of the 2D image; t is the distance from the center point in the 2D image to the left border of the 2D image; and d is the distance from the center point in the 2D image to the right border of the 2D image.

[0127] In one example, the radius of the Gaussian circle described above can be determined based on the following steps: Assuming the preset overlap threshold is 0.5, it should be noted that the overlap threshold can be set based on the user's experience or the recognition accuracy of the target detection model.

[0128] (1) Calculate a, b, and c respectively according to the following formulas:

[0129] Then calculate r1 using the following formula:

[0130] (2) Calculate a, b, and c respectively according to the following formulas:

[0131] Then calculate r2 using the following formula:

[0132] (3) Calculate a, b, and c respectively according to the following formulas.

[0133] Then calculate r3 according to the following formula:

[0134] Calculate the radius r of the Gaussian circle based on r1, r2, and r3: r = min(r1, r2, r3) It should be noted that a, b, and c above are all intermediate variables involved in the calculation of the radius of the Gaussian circle.

[0135] If the Gaussian circle is a Gaussian ellipse, the radius of the Gaussian ellipse, r_w, r_h, also needs to be calculated based on the following formulas: r_w = sqrt(w / h) r r_h = sqrt(h / w) r Where r_w is the major radius of the Gaussian ellipse, r_h is the minor radius of the Gaussian ellipse, and sqrt is the square root function of a non-negative real number.

[0136] In one example, generating a heatmap corresponding to the radius of the Gaussian ellipse may include: calculating the horizontal and vertical diameters of the ellipse, respectively, based on the radius (r_w, r_h) of the Gaussian ellipse on the feature map of the target object obtained above, with d_w=2. r_w+1,d_h=2 r_h+1. Assume the standard deviations of the Gaussian functions in the horizontal and vertical directions of the ellipse are sigma_w = d_w / 6 and sigma_h = d_h / 6, respectively. The heatmap h(x, y) of the target object can be calculated based on the preset Gaussian functions:

[0137] It should be noted that "sigma" represents the standard deviation, which is used to smooth images to reduce image noise.

[0138] In this implementation, additional image smoothing processing can be added, or the image can be left unprocessed; both options are valid and fall within the scope of protection of this application.

[0139] In this implementation, a heat map can be generated by the distance between the border containing the target object and the center point of the border in a two-dimensional image, thereby generating a more accurate heat map using the corrected size information.

[0140] like Figure 5 The flowchart 500 illustrates an embodiment of a target detection method according to this application, which may include the following steps: Step 501: Obtain the image to be tested.

[0141] In this embodiment, the execution body of the target detection method (e.g.) Figure 1 The terminal devices 101 and 102 (or server 104) can retrieve the image to be tested from their storage or receive an externally input image to be tested, such as an image input by a user or an image transmitted from other devices. The image to be tested can be an image to be predicted by a pre-trained object detection model.

[0142] Step 502: Input the image to be tested into the pre-trained target detection model to obtain the output result; In this embodiment, the aforementioned execution entity can, according to Figure 2 , Figure 4 The target detection model in the image identifies the target object and determines its category and location.

[0143] In one example, the aforementioned entity inputs the image to be tested into the target detection model to achieve recognition of the image.

[0144] In one example, the recognition of the image to be tested can be achieved by inputting the image to be tested from step 501 into the target detection model in step 502 to obtain the output result of the image to be tested. The image to be tested can be one or more images, or one or more frames from a video.

[0145] Step 503: Based on the output results, determine the category and location of the target object in the image to be tested.

[0146] In this embodiment, the execution entity can determine the category and location of the target object in the image to be tested based on the output result of the target detection model in step 502, so as to complete the recognition of the image to be tested.

[0147] It should be noted that by inputting the image to be tested into the target detection model in step 502, the category and location of the target object in the image to be tested can be obtained, as well as the pose of the target object in the image to be tested.

[0148] In this embodiment, target detection can involve 3D target detection and 2D target detection; 3D target detection can be performed using monocular cameras, binocular cameras, multi-target cameras, etc. The task of 3D target detection using a monocular camera can be accomplished through a monocular 3D target detection model. Due to the low cost and better economic efficiency of monocular cameras, monocular 3D target detection models are widely used, especially in the field of autonomous driving navigation, where they can support autonomous driving decisions for vehicles. By inputting a single RGB image into the monocular 3D target detection model, the model can predict the category and location information of target objects in the image. This location information can include the height h, width w, and center point coordinates (x, y) of the target object.

[0149] In this embodiment, the target detection method 500 first acquires the image to be tested; then, using the aforementioned target detection model, while ensuring the high accuracy of the target detection model, it identifies the image to be tested to obtain the category and location of the target object in the image to be tested.

[0150] In some optional implementations of this embodiment, determining the category and location of the target object in the image under test based on the output results includes: determining the category of the target object in the image under test based on the classification processing results included in the output results; and determining the location of the target object in the image under test based on the regression processing results included in the output results.

[0151] In this implementation, the aforementioned execution entity can obtain the category of the target object in the image under test corresponding to the classification processing result, and the location of the target object in the image under test corresponding to the regression processing result, based on the classification processing result and regression processing result included in the output result.

[0152] Further reference Figure 6 As an implementation of the methods shown in the above figures, this application discloses an embodiment of an apparatus for generating a target detection model, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0153] like Figure 6As shown, embodiments of this application provide an apparatus 600 for generating a target detection model. The apparatus 600 includes: an image acquisition module 601, a feature map acquisition module 602, a heat map generation module 603, a first determination module 604, a second determination module 605, and a model acquisition module 606. The system includes: an image acquisition module 601 configured to acquire a target sample image; a feature map acquisition module 602 configured to input the target sample image into the backbone network of the machine learning model to obtain a feature map of the target sample image; a heatmap generation module 603 configured to process the target sample image and the feature map to generate a heatmap; a first determination module 604 configured to classify each location point in the heatmap using the classification network of the machine learning model, and determine a classification loss value based on the classification results corresponding to each location point and the category labels of each location point in the heatmap; a second determination module 605 configured to perform regression processing on each location point in the heatmap using the regression network of the machine learning model, and determine a regression loss value based on the regression results corresponding to each location point and the regression labels of each location point in the heatmap; and a model acquisition module 606 configured to train the machine learning model based on the classification loss value and the regression loss value to obtain a target detection model.

[0154] In this embodiment, the specific processing and technical effects of the image acquisition module 601, feature map acquisition module 602, heatmap generation module 603, first determination module 604, second determination module 605, and model acquisition module 606 in the device 600 for generating the target detection model can be found in the following references. Figure 2 The corresponding steps are 201 to 206 in the embodiment. Optionally, the first determining module and the second determining module can be the same or different modules.

[0155] In some embodiments, the heatmap generation module 603 includes: an image projection unit configured to project the bounding box containing the target object in the target sample image onto a feature map to obtain a two-dimensional image; a radius determination unit configured to determine the radius of a Gaussian circle based on the target position information; and a heatmap determination unit configured to determine a heatmap based on the radius of the Gaussian circle.

[0156] In some embodiments, the target location information includes target size information; the heatmap determination unit is further configured to: determine the radius of a Gaussian circle based on a preset Gaussian function and a preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box of the target object in the target sample image and the bounding box of the target object in the two-dimensional image; and determine a heatmap based on the radius of the Gaussian circle and the target size information.

[0157] In some embodiments, the target object in the target sample image is located on the boundary of the feature map, and the target position information includes target coordinate information and target size information; the apparatus for generating the target detection model further includes: a distance determination module, configured to determine the distance information between the center point of the bounding box where the target object is located in the two-dimensional image and the bounding box where the target object is located in the two-dimensional image based on the target coordinate information; and a size determination module, configured to determine the corrected size information based on the distance information and the target size information.

[0158] In some embodiments, the heatmap determination unit is further configured to: determine the radius of a Gaussian circle based on a preset Gaussian function and a preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box containing the target object in the target sample image and the bounding box containing the target object in the two-dimensional image; and determine the heatmap based on the radius of the Gaussian circle and the corrected size information.

[0159] In some embodiments, each location point within the heatmap includes: a location point corresponding to each target sample image within the radius of a Gaussian circle centered on the coordinate information of the center point of the target object.

[0160] In some embodiments, the model obtaining module 606 is further configured to: normalize the regression loss value and the classification loss value to obtain the regression loss value and the classification loss value corresponding to each location point; train the machine learning model based on the classification loss value and the corresponding first preset weight, and the regression loss value and the corresponding second preset weight corresponding to each location point to obtain the object detection model.

[0161] Further reference Figure 7 As an implementation of the methods shown in the above figures, this application discloses an embodiment of a target detection device, which is similar to... Figure 5 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0162] like Figure 7 As shown, an embodiment of this application provides a target detection device 700, which includes an image acquisition module 701, an image input module 702, and a result determination module 703. The image acquisition module 701 is configured to acquire a test image; the image input module 702 is configured to input the test image into a pre-trained target detection model to obtain an output result; and the result determination module 703 is configured to determine the category and location of the target object in the test image based on the output result.

[0163] In this embodiment, the specific processing of the image acquisition module 701, the image input module 702, and the result determination module 703 in the target detection device 700, and the resulting technical effects, can be referred to respectively. Figure 5 Steps 501 to 503 in the corresponding embodiment.

[0164] In some embodiments, the result determination module 703 is further configured to: determine the category of the target object in the image under test based on the classification processing result included in the output result; and determine the location of the target object in the image under test based on the regression processing result included in the output result.

[0165] The target detection device provided in the embodiments of this application acquires the image to be tested through the image acquisition module 701; then, through the image input module 702, it uses the above-mentioned target detection model to identify the image to be tested while ensuring the high accuracy of the target detection model, so as to obtain the category and location of the target object in the image to be tested.

[0166] like Figure 8 The diagram shown is a block diagram of an electronic device for generating a target detection model or a target detection method according to embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0167] like Figure 8 As shown, the electronic device includes one or more processors 801, a memory 802, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take the 801 processor as an example.

[0168] The memory 802 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the method for generating a target detection model or the target detection method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the method for generating a target detection model or the target detection method provided in this application.

[0169] Memory 802, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the method for generating a target detection model or the target detection method in the embodiments of this application (e.g., attached). Figure 6 The image acquisition module 601, feature map acquisition module 602, heatmap generation module 603, first determination module 604, second determination module 605, and model acquisition module 606 shown are also included; or Figure 7 The image acquisition module 701, image input module 702, and result determination module 703 are shown. The processor 801 executes various server functions and data processing by running non-transient software programs, instructions, and modules stored in the memory 802, that is, it implements the method for generating a target detection model or the target detection method in the above method embodiments.

[0170] The memory 802 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the blockchain-based information processing electronic device. Furthermore, the memory 802 may include high-speed random access memory and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 802 may optionally include memory remotely located relative to the processor 801, and these remote memories can be connected to the blockchain-based information processing electronic device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0171] The electronic device for generating a target detection model or a target detection method may further include an input device 803 and an output device 804. The processor 801, memory 802, input device 803, and output device 804 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0172] Input device 803 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of blockchain-based information processing electronic devices, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 804 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0173] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0176] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0177] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0178] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0179] In embodiments of this application, an autonomous vehicle is provided, which may include a device for generating a target detection model and / or a target detection device.

[0180] The following is based on Figure 9 and Figure 10 An example is provided for illustration.

[0181] Figure 9 This is a structural schematic diagram of one embodiment of an autonomous vehicle according to this application. Figure 9As shown, the autonomous vehicle 900 includes a device 600 for generating a target detection model.

[0182] An autonomous vehicle 900 is equipped with at least one sensor for acquiring target sample images, in order to train a target detection model based on the acquired target sample images, etc. For example, the sensor installed in the autonomous vehicle may be one or more of a camera, LiDAR, or millimeter-wave radar.

[0183] In some embodiments, the apparatus 600 for generating an object detection model is configured to: acquire a target sample image; input the target sample image into a backbone network included in a machine learning model to obtain a feature map of the target sample image; process the target sample image and the feature map to generate a heatmap; classify each location point in the heatmap using a classification network included in the machine learning model, and determine a classification loss value based on the classification result corresponding to each location point and the category label of each location point in the heatmap; perform regression processing on each location point in the heatmap using a regression network included in the machine learning model, and determine a regression loss value based on the regression result corresponding to each location point and the regression label of each location point in the heatmap; and train the machine learning model based on the classification loss value and the regression loss value to obtain an object detection model.

[0184] The method for generating object detection models in the above embodiments can obtain continuous classification labels and corresponding classification loss values ​​based on each position in the heatmap generated by processing the target sample image and feature map, and determine the corresponding regression loss values ​​based on continuous regression labels, which can achieve global optimization. In addition, it can also solve the model training instability caused by the discrete label loss calculation method under hard threshold, so as to significantly improve the performance of the object detection model.

[0185] Figure 10 This is a structural schematic diagram of one embodiment of an autonomous vehicle according to this application. Figure 10 As shown, the autonomous vehicle 1000 includes a target detection device 700.

[0186] An autonomous vehicle 1000 is equipped with at least one sensor for acquiring images to be tested, in order to perform target detection, etc., based on the acquired images. For example, the sensor installed in the autonomous vehicle may be one or more of a camera, LiDAR, or millimeter-wave radar.

[0187] In some embodiments, the target detection device 700 is configured to acquire a test image; input the test image into a pre-trained target detection model to obtain an output result; and determine the category and location of the target object in the test image based on the output result.

[0188] The target detection method described in the above embodiments can identify the target image while ensuring the high accuracy of the target detection model, so as to obtain the category and location of the target object in the target image.

[0189] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for generating an object detection model, comprising: Acquire a target sample image, wherein the target sample image is a three-dimensional image or a two-dimensional image; The target sample image is input into the backbone network of the machine learning model to obtain the feature map of the target sample image; The target sample image and the feature map are processed to generate a heatmap; The classification network included in the machine learning model is used to classify each location point in the heatmap, and the classification loss value is determined based on the classification results corresponding to each location point and the category labels of each location point in the heatmap. The regression network included in the machine learning model is used to perform regression processing on each location point in the heatmap, and the regression loss value is determined based on the regression processing result corresponding to each location point and the regression label of each location point in the heatmap. The machine learning model is trained based on the classification loss value and the regression loss value to obtain the target detection model; The step of processing the target sample image and the feature map to generate a heatmap includes: Based on the calibration parameters of the camera that captured the target sample image, a coordinate transformation matrix is ​​determined. Based on the coordinate transformation matrix, the bounding box of the target object in the target sample image is projected onto the feature map to obtain a two-dimensional image. When the target object is on the boundary of the feature map, the outermost two-dimensional bounding box of the three-dimensional projection box on the feature map exceeds the boundary of the feature map, and the outermost two-dimensional bounding box is truncated by the boundary of the feature map. The radius of the Gaussian circle is determined based on the target position information of the border containing the target object in the two-dimensional image. The heat map is determined based on the radius of the Gaussian circle.

2. The method according to claim 1, wherein, The target location information includes target size information; Determining the heatmap based on the radius of the Gaussian circle includes: The radius of the Gaussian circle is determined based on the target size information and the preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box of the target object in the target sample image and the bounding box of the two-dimensional image. The heatmap is determined based on the radius of the Gaussian circle and a preset Gaussian function.

3. The method according to claim 1, wherein, The target object in the target sample image is located on the boundary of the feature map, and the target location information includes target coordinate information and target size information; the method further includes: Based on the target coordinate information, determine the distance information between the center point of the border containing the target object in the two-dimensional image and the border containing the target object in the two-dimensional image; Based on the distance information and the target size information, the corrected size information is determined.

4. The method according to claim 3, wherein, Determining the heatmap based on the radius of the Gaussian circle includes: The radius of the Gaussian circle is determined based on a preset Gaussian function and a preset overlap region threshold, wherein the preset overlap region threshold is used to characterize the overlap threshold between the bounding box containing the target object in the target sample image and the bounding box containing the target object in the two-dimensional image. The heat map is determined based on the radius of the Gaussian circle and the corrected size information.

5. The method according to any one of claims 1-4, wherein, Each location point within the heatmap includes: a location point corresponding to each target sample image within the radius of the Gaussian circle, centered on the coordinate information of the center point of the target object.

6. The method according to claim 5, wherein, The step of training the machine learning model based on the classification loss value and the regression loss value to obtain the object detection model includes: The regression loss value and the classification loss value are normalized to obtain the regression loss value and the classification loss value corresponding to each location point. The machine learning model is trained based on the classification loss value and the corresponding first preset weight for each location point, as well as the regression loss value and the corresponding second preset weight for each location point, to obtain the object detection model.

7. A target detection method, comprising: Acquire the image to be tested; The image to be tested is input into the target detection model generated by the method described in any one of claims 1 to 6 to obtain the output result; Based on the output results, the category and location of the target object in the image under test are determined.

8. The method according to claim 7, wherein, The step of determining the category and location of the target object in the image under test based on the output result includes: Based on the classification processing results included in the output, the category of the target object in the image to be tested is determined; The location of the target object in the image under test is determined based on the regression processing results included in the output.

9. An apparatus for generating a target detection model, comprising: The image acquisition module is configured to acquire a target sample image, wherein the target sample image is a three-dimensional image or a two-dimensional image; The feature map acquisition module is configured to input the target sample image into the backbone network of the machine learning model to obtain the feature map of the target sample image; The heatmap generation module is configured to process the target sample image and the feature map to generate a heatmap; The first determining module is configured to classify each location point in the heatmap using the classification network included in the machine learning model, and determine the classification loss value corresponding to each location point based on the classification result corresponding to each location point and the category label of each location point in the heatmap. The second determining module is configured to perform regression processing on each location point in the heatmap through the regression network included in the machine learning model, and determine the regression loss value based on the regression processing result corresponding to each location point and the regression label of each location point in the heatmap. The model acquisition module is configured to train the machine learning model based on the classification loss value and the regression loss value to obtain an object detection model; The heatmap generation module includes: An image projection unit is configured to determine a coordinate transformation matrix based on the calibration parameters of the camera that captures the target sample image, and to project the bounding box of the target object in the target sample image onto the feature map based on the coordinate transformation matrix to obtain a two-dimensional image. When the target object is on the boundary of the feature map, the outermost two-dimensional bounding box of the three-dimensional projection box on the feature map exceeds the boundary of the feature map, and the outermost two-dimensional bounding box is truncated by the boundary of the feature map. The radius determination unit is configured to determine the radius of the Gaussian circle based on the target position information; The heatmap determination unit is configured to determine the heatmap based on the radius of the Gaussian circle.

10. The apparatus according to claim 9, wherein, The target object in the target sample image is located on the boundary of the feature map, and the target location information includes target coordinate information and target size information; The device further includes: The distance determination module is configured to determine the distance information between the center point of the border containing the target object in the two-dimensional image and the border containing the target object in the two-dimensional image, based on the target coordinate information. The size determination module is configured to determine the corrected size information based on the distance information and the target size information.

11. A target detection device, comprising: The image acquisition module is configured to acquire the image to be tested; An image input module is configured to input the image to be tested into a target detection model generated by the method described in any one of claims 1 to 6, and obtain an output result; The result determination module is configured to determine the category and location of the target object in the image under test based on the output result.

12. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein a computer program is stored thereon, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

14. An autonomous vehicle, comprising: The apparatus for generating a target detection model as described in claim 9 and / or the target detection apparatus as described in claim 11.

Images