Target detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a target detection method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: performing feature extraction on a target image based on a backbone network of a Center Net model to obtain a first feature map; generating a thermodynamic diagram capable of representing central point distribution of a target object included in the target image based on the first feature map and an elliptical Gaussian kernel; determining a detection result of the target object based on the thermodynamic diagram; wherein the long-axis kernel radius and the short-axis kernel radius of the elliptical Gaussian kernel are determined based on a circumscribed rectangle of central point distribution of the target object, the size of the circumscribed rectangle and the intersection-to-union ratio have a specific relationship, and the intersection-to-union ratio is the ratio of the intersection area and union area of a prediction detection frame and a real detection frame of the target object. According to the method and the device, diversified detection requirements can be met, and particularly, the method and the device have higher detection precision and adaptivity for target objects with relatively large length-width ratio differences.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target detection method, device, electronic device and storage medium. Background Art

[0002] CenterNet is a target detection model that determines the location of a target object by predicting its center point. It has shown high detection speed and accuracy in the field of computer vision, especially in target detection tasks. Moreover, in CenterNet, Gaussian Kernel plays a crucial role, which is mainly used to generate heatmaps, thereby helping the CenterNet model locate and identify the center point of the target in the image.

[0003] In related technologies, a circular Gaussian kernel is usually used to generate a heatmap to help the CenterNet model locate and identify the center point of the target object in the image. This method works well when dealing with objects with regular shapes and consistent proportions, because it can represent the confidence of the position of the center point of the target object and its surrounding area in a smooth and continuous manner. However, for objects with large differences in aspect ratios, such as people and vehicles, the circular Gaussian kernel may not be able to fully and accurately reflect the actual shape of the object. This is because the use of the circular Gaussian kernel in the CenterNet model is based on the assumption that the influence area of ​​the center point of the target object can be approximated as a circle or a nearly circular area, and for objects with extreme aspect ratios, this assumption may no longer be fully applicable, resulting in limited detection accuracy. Summary of the invention

[0004] In response to the above technical problems, the embodiments of the present application provide a target detection method, device, electronic device and storage medium, which can meet diverse detection needs, especially for target objects with large differences in aspect ratios, and have higher detection accuracy and adaptability.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a target detection method, comprising:

[0007] Based on the backbone network of the CenterNet model, features of the target image are extracted to obtain the first feature map;

[0008] Based on the first feature map and the elliptical Gaussian kernel, a heat map capable of characterizing the distribution of center points of the target object included in the target image is generated;

[0009] Based on the heat map, determine the detection result of the target object;

[0010] Among them, the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object. There is a specific relationship between the size of the circumscribed rectangle and the intersection-and-union ratio. The intersection-and-union ratio is the ratio of the intersection area to the union area of ​​the predicted detection box and the true detection box of the target object.

[0011] In some embodiments, the specific relationship between the size of the bounding rectangle and the intersection-over-union ratio is:

[0012]

[0013] Among them, w represents the width of the circumscribed rectangle, h represents the length of the circumscribed rectangle, iou represents the intersection-union ratio, and W TP Indicates the width of the real detection box, H TP Indicates the length of the real detection box.

[0014] In some embodiments, the backbone network of the CenterNet model is composed of a target residual network and a feature pyramid network.

[0015] In some embodiments, the target residual network includes a convolutional layer and at least one residual block layer, each residual block layer includes at least one residual block, and each residual block includes at least one convolutional layer.

[0016] In some embodiments, a backbone network based on a CenterNet model performs feature extraction on a target image to obtain a first feature map, including:

[0017] Based on the target residual network, the target image is subjected to feature extraction to obtain feature maps of multiple different scales;

[0018] Based on the feature pyramid network, multiple feature maps of different scales are fused to obtain the first feature map.

[0019] In some embodiments, a plurality of feature maps of different scales are fused based on a feature pyramid network to obtain a first feature map, including:

[0020] Based on the feature pyramid network, multiple feature maps of different scales are fused to obtain a second feature map;

[0021] The second feature map is upsampled based on the deconvolution layer in the CenterNet model to obtain the first feature map.

[0022] In some embodiments, based on the heat map, determining the detection result of the target object includes:

[0023] Based on the first feature map, determining the size of the target object and the offset of the center point coordinates of the target object respectively;

[0024] The detection result of the target object is determined based on the heat map, the size of the target object and the offset of the center point coordinates of the target object.

[0025] In a second aspect, an embodiment of the present application provides a target detection device, including:

[0026] A feature extraction module is used to extract features of the target image based on the backbone network of the CenterNet model to obtain a first feature map;

[0027] A generating module, configured to generate a heat map capable of representing the distribution of center points of a target object included in the target image based on the first feature map and the elliptical Gaussian kernel;

[0028] A determination module, used to determine the detection result of the target object based on the heat map;

[0029] Among them, the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object. There is a specific relationship between the size of the circumscribed rectangle and the intersection-and-union ratio. The intersection-and-union ratio is the ratio of the intersection area to the union area of ​​the predicted detection box and the true detection box of the target object.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is used to store executable data instructions; when the processor is used to execute the executable data instructions stored in the memory, the steps in the target detection method of the first aspect are implemented.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the target detection method of the first aspect are implemented.

[0032] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the target detection method of the first aspect.

[0033] The target detection method, device, electronic device and storage medium provided in the embodiment of the present application first extract features of the target image based on the backbone network of the CenterNet model to obtain a first feature map, and then generate a heat map that can characterize the distribution of the center points of the target object included in the target image based on the first feature map and the elliptical Gaussian kernel, and then determine the detection result of the target object based on the heat map. In this way, the embodiment of the present application generates a heat map based on the elliptical Gaussian kernel, and the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object, and there is a specific relationship between the size of the circumscribed rectangle and the intersection and union ratio, that is, based on the specific relationship, the circumscribed rectangle of the center point distribution of the target object can be determined, and then the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined by the circumscribed rectangle, so that the elliptical Gaussian kernel can accurately reflect the actual shape of the target object, especially for target objects with large differences in aspect ratio, with higher detection accuracy and adaptability, meeting diverse detection needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 One of the flowcharts of a target detection method provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of center point distribution when the intersection-over-union ratio is 0.7 provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of center point distribution when the intersection-over-union ratio is 0.8 provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of the structure of a backbone network of a CenterNet model provided in an embodiment of the present application;

[0039] Figure 5 One of the structural schematic diagrams of a convolutional neural network basic block provided in an embodiment of the present application;

[0040] Figure 6 A second structural diagram of a convolutional neural network basic block provided in an embodiment of the present application;

[0041] Figure 7A second flow chart of a target detection method provided in an embodiment of the present application;

[0042] Figure 8 A third flow chart of a target detection method provided in an embodiment of the present application;

[0043] Fig. 9 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0044] Fig.10 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] It should be noted that in the description of the embodiments of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0047] In order to facilitate a clearer understanding of the embodiments of the present application, some related technologies are first introduced as follows.

[0048] The current mainstream target detection methods include the following two:

[0049] (I) Anchor-based detection methods. This type of method is easy to implement, but has poor target adaptability and requires adjustment of the hyperparameters of the pre-selected box. Taking the YOLO (You Only Look Once) algorithm as an example, the YOLO algorithm has shown significant advantages in the field of computer vision, especially in target detection tasks, due to its simplicity of implementation and strong portability. The YOLO algorithm can predict the location and category of the target in the image through a single forward propagation, which greatly improves the detection speed and makes it an ideal choice for real-time multi-target detection. However, in certain scenarios, especially when facing obstacles and daily objects with similar color features and regular geometric shapes (such as fire hydrants, traffic cones, billboards, etc.), the YOLO algorithm is prone to false detection. This is because the features of these objects are similar to the shape of the preset anchor box, making it difficult to effectively distinguish them through traditional filtering mechanisms. Although the image samples of obstacles and daily objects can be expanded through data enhancement to reduce the false detection rate, this method also has its inherent limitations while introducing new features.

[0050] (II) Anchor-free detection method. The CenterNet algorithm is its representative algorithm. CenterNet determines the shape and position of an object by predicting the center point of the object as well as its width and height, combined with the probability distribution model of the Gaussian kernel. This method does not require manual design of pre-selection boxes and can adapt to objects of different sizes to be detected, thereby significantly improving the flexibility and accuracy of the detection algorithm.

[0051] However, in the traditional CenterNet model, the Gaussian kernel uses a circle with the predicted center point as the center, and the Gaussian distribution probability of the pixels within its radius is used to evaluate the accuracy of the center point. This method works well when processing objects with regular shapes and consistent proportions, but for objects with large differences in aspect ratios, such as people and vehicles, the circular Gaussian kernel may not be able to fully and accurately reflect the actual shape of the object, resulting in limited detection accuracy.

[0052] In order to overcome at least some of the defects in the related art, the embodiments of the present application provide a target detection method, device, electronic device and storage medium, which can meet diverse detection needs, especially for target objects with large differences in aspect ratios, and have higher detection accuracy and adaptability.

[0053] The target detection method, device, electronic device and storage medium provided in the embodiments of the present application are exemplarily introduced below in conjunction with the drawings in the embodiments of the present application.

[0054] Figure 1 One of the flow charts of a target detection method provided in an embodiment of the present application is as follows: Figure 1 As shown, the method includes:

[0055] S101. Perform feature extraction on the target image based on the backbone network of the CenterNet model to obtain a first feature map.

[0056] It should be noted that the execution subject of the target detection method provided in the embodiment of the present application can be an electronic device, a component in an electronic device, an integrated circuit or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a wearable device, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit this.

[0057] The following takes a computer executing the target detection method provided in the embodiment of the present application as an example to describe in detail the technical solution of the embodiment of the present application.

[0058] It should be noted that the target image in the embodiments of the present application can be any image containing the target object to be detected, which can come from various scenes. For example, (1) daily life images: including photos taken by people in daily life, such as landscape photos, portrait photos, family activity photos, etc. These images may include targets such as people, animals, vehicles, buildings, etc. that need to be detected. (2) surveillance video frames: single-frame images extracted from videos captured by surveillance cameras, used for security monitoring, crowd counting, abnormal behavior detection, etc. These images may include targets such as pedestrians, vehicles, animals, etc. (3) satellite images: images of the earth's surface taken by satellites, used in the fields of geographic information systems, environmental monitoring, urban planning, etc. These images may include targets such as roads, buildings, water bodies, forests, etc. that need to be detected. (4) industrial images: images captured on industrial production lines, used for product quality inspection, defect identification, etc. For example, in the manufacturing industry, scratches, cracks or foreign objects on products are detected. (5) traffic images: including surveillance images of traffic intersections, images taken by vehicle-mounted cameras, etc., used for traffic flow analysis, violation detection, etc. These images may contain objects such as vehicles, pedestrians, and traffic signs. (6) Agricultural images: images used for agricultural production, such as crop growth monitoring and pest and disease identification. By detecting the color, shape, texture and other features of crop leaves, the growth status and health of the crop can be judged.

[0059] It should be noted that the backbone network of the CenterNet model is one of its core components, responsible for extracting features from the input image and providing a basis for subsequent center point prediction and size prediction. The backbone network of the CenterNet model can be an hourglass network (Hourglass Network), a deep layer aggregation network (Deep Layer Aggregation Network, DLANet) and a residual network (Residual Network, ResNet), etc. These network structures have their own characteristics and can be selected and adjusted according to different application requirements.

[0060] In an embodiment of the present application, after obtaining a target image to be detected, feature extraction can be performed on the target image based on the backbone network of the CenterNet model to obtain a first feature map.

[0061] S102, generating a heat map capable of characterizing the distribution of center points of a target object included in the target image based on the first feature map and the elliptical Gaussian kernel;

[0062] Among them, the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object, and there is a specific relationship between the size of the circumscribed rectangle and the intersection-and-union ratio, and the intersection-and-union ratio is the ratio of the intersection area to the union area of ​​the predicted detection box and the real detection box of the target object.

[0063] It should be noted that the CenterNet model predicts the position of the target object (object) in the target image by generating a heat map. Each pixel in the heat map corresponds to the center point of an object of a specific category, and its value reflects the possibility of the pixel becoming the center of the object. In order to improve the accuracy of detection, CenterNet uses a Gaussian kernel to represent the confidence of the center point of the object, where the radius of the Gaussian kernel is related to the size and shape of the object. However, the traditional CenterNet model uses a circular Gaussian kernel. For objects with large differences in aspect ratios, the circular Gaussian kernel may not accurately reflect the actual shape of the object, resulting in limited detection accuracy. In response to this defect, the embodiment of the present application experimentally analyzes the distribution of the center point of the target object as the intersection over Union (IoU) of the predicted detection box (bounding box) of the target object changes with the true detection box (ground truth). The experimental results show that when the IoU is low, the center point distribution is star-shaped; when the IoU is close to 0.72, the center point distribution changes to a diamond shape; as the IoU continues to increase, the center point distribution gradually changes to an ellipse, and finally when the IoU is equal to 1, the center point distribution approaches a circle.

[0064] For example, Figure 2 A schematic diagram of center point distribution when the intersection-over-union ratio is 0.7 provided in an embodiment of the present application, Figure 3 This is a schematic diagram of the center point distribution when the intersection-over-combination ratio is 0.8 provided in an embodiment of the present application. Figure 2 As shown in , when the intersection-over-union ratio is 0.7, the center point distribution is star-shaped, and at this time it is close to a diamond shape; Figure 3 As shown in the figure, when the intersection-over-union ratio is 0.8, the center point distribution is elliptical. Moreover, it should be noted that the aspect ratio of the bounding rectangle of the figure formed by the set of center points of the detection frame (i.e., the bounding rectangle of the target object's center point distribution) is similar to that of the real detection frame, and there is a specific relationship between the size of the bounding rectangle and the IoU value.

[0065] In some embodiments, the specific relationship between the size of the bounding rectangle and the intersection-over-union ratio is:

[0066]

[0067] Wherein, w represents the width of the circumscribed rectangle, h represents the length of the circumscribed rectangle, iou represents the intersection-and-union ratio, and W TP Represents the width of the real detection box, H TP Represents the length of the real detection box.

[0068] In some embodiments, the size of the circumscribed rectangle can be determined based on a specific relationship between the size of the circumscribed rectangle and the intersection-and-overlap ratio, and the length of the circumscribed rectangle can be used as the major axis kernel radius of the elliptical Gaussian kernel, and the width of the circumscribed rectangle can be used as the minor axis kernel radius of the elliptical Gaussian kernel.

[0069] In an embodiment of the present application, after determining the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel based on the size of the circumscribed rectangle, a heat map capable of characterizing the distribution of center points of the target object included in the target image can be generated based on the first feature map and the elliptical Gaussian kernel.

[0070] It should be noted that the target object in the embodiment of the present application is an object to be detected, which can be one object or multiple objects.

[0071] S103: Determine a detection result of the target object based on the heat map.

[0072] In some embodiments, after obtaining the heat map, the local maximum point (or peak point) representing the center point of the target object can be identified from the heat map, and this point can be used as the center point of the target object, and then a detection box of the target object is generated based on the center point of the target object as the detection result of the target object.

[0073] In some embodiments, non-maximum suppression (NMS) or a similar algorithm may be used to identify a peak point representing a center point of a target object.

[0074] In some embodiments, the size of the target object (such as width and height) and the offset of the center point (used to more accurately locate the center point) can be predicted based on the CenterNet model, and then these predictions can be combined with the peak points of the heat map to generate a detection box of the target object as the detection result of the target object.

[0075] It can be understood that the target detection method provided in the embodiment of the present application first extracts features from the target image based on the backbone network of the CenterNet model to obtain a first feature map, and then generates a heat map capable of characterizing the distribution of the center points of the target object included in the target image based on the first feature map and the elliptical Gaussian kernel, and then determines the detection result of the target object based on the heat map. In this way, the embodiment of the present application generates a heat map based on the elliptical Gaussian kernel, and the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object, and there is a specific relationship between the size of the circumscribed rectangle and the intersection and union ratio, that is, based on the specific relationship, the circumscribed rectangle of the center point distribution of the target object can be determined, and then the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined by the circumscribed rectangle, so that the elliptical Gaussian kernel can accurately reflect the actual shape of the target object, especially for target objects with large differences in aspect ratio, with higher detection accuracy and adaptability, meeting diverse detection needs.

[0076] In some embodiments, the backbone network of the CenterNet model is composed of a target residual network and a feature pyramid network.

[0077] It should be noted that the residual network ResNet is a deep neural network architecture. Its core idea is to solve the difficulties in deep neural network training by introducing "residual connection". In traditional neural networks, as the number of layers increases, training deep networks often encounters problems of gradient vanishing or gradient exploding, resulting in decreased network performance. The residual network uses residual connections to allow input to be passed directly to output, thereby effectively alleviating these problems.

[0078] It should be noted that the target residual network in the embodiment of the present application can be a traditional ResNet or other network versions of the traditional ResNet, such as ResNet-18, ResNet-34, ResNet-50, ResNet-101 and ResNet-152, etc. The embodiment of the present application does not make specific limitations on this and can be selected based on the adaptability of the actual application.

[0079] It should be noted that Feature Pyramid Network (FPN) is also a deep neural network architecture. Its main purpose is to extract and fuse feature information of images at different scales by constructing a feature pyramid, thereby improving the accuracy of detection.

[0080] In some embodiments, the implementation of FPN includes: (1) Bottom-up path: This is the forward propagation process of the network. A convolutional neural network (such as ResNet) can be used as the backbone network to generate a series of feature maps of different scales. These feature maps have different resolutions and semantic information. (2) Top-down path: Through upsampling operations, the resolution of high-level feature maps is gradually enlarged to match the resolution of low-level feature maps. At the same time, through lateral connections, the upsampled high-level feature maps are fused with the low-level feature maps of the corresponding scale. The fused feature maps contain both high-level semantic information and low-level spatial information. These fused feature maps constitute different levels of the feature pyramid.

[0081] It can be understood that the embodiment of the present application combines the target residual network and the feature pyramid network to form the backbone network of the CenterNet model, which can not only solve the problem of gradient disappearance or gradient explosion, but also enable the CenterNet model to better handle target objects of different sizes and scales, thereby improving the detection performance of the CenterNet model.

[0082] In some embodiments, the target residual network includes a convolutional layer and at least one residual block layer, each of the residual block layers includes at least one residual block, and each of the residual blocks includes at least one convolutional layer.

[0083] It should be noted that the residual block in the residual network is its basic component unit. In the embodiment of the present application, each residual block may include at least one convolutional layer. Among them, the convolution kernel size and step size of the convolutional layer included in the target residual network and at least one convolutional layer included in each residual block can be adaptively set based on the actual application, and the embodiment of the present application does not specifically limit this.

[0084] For example, Figure 4 A schematic diagram of the structure of a backbone network of a CenterNet model provided in an embodiment of the present application is shown in FIG. Figure 4As shown in the figure, the network structure includes a convolution layer conv1, four residual block layers Layer1, Layer2, Layer3, Layer4, and output layer heads, where the output layer heads includes three output heads hm, wh, and reg; hm represents the heat map, which is used to predict the center point of the target object; wh represents the loss of the target object size (length and width), which is used to constrain the detection box size of the target object; reg represents the center point offset of the target object. The center point in the heat map is located on the grid line, and the offset represents the distance from the point in the grid to the edge of the grid line. Among them, Layer1 and Layer2 are composed of 2 convolutional neural network basic blocks CBR and batch normalization (Batch Normalization, BN) layers, Layer3 is composed of 6 convolutional neural network basic blocks CBR and batch normalization BN layers, and Layer4 is composed of 4 convolutional neural network basic blocks CBR and batch normalization BN layers.

[0085] It should be noted that CBR is a layer combination pattern in a convolutional neural network, including: Convolution layer: used to extract features of input data. Through sliding windows and dot product operations, the convolution layer can generate feature maps that reflect the different features of the input data. Batch Normalization layer: used to improve the training speed and stability of the neural network. Batch normalization normalizes each small batch of data so that the input data of each layer has a similar distribution, which helps to speed up the training process and reduce the risk of overfitting. Linear Rectification Function Layer (ReLU): As an activation function, the ReLU layer sets negative values ​​to zero and keeps positive values ​​unchanged. This nonlinear transformation helps the neural network learn complex feature representations.

[0086] For example, Figure 5 One of the structural diagrams of a convolutional neural network basic block provided in an embodiment of the present application is as follows: Figure 5 As shown in the figure, this structure is the typical structure of the convolutional neural network basic block CBR, including a convolution layer with a convolution kernel size of 3×3, a batch normalization layer, and a ReLU activation layer.

[0087] In some embodiments, Figure 4 The convolution layer conv1 in has 3 input channels, 64 output channels, a convolution kernel size of 4×4, and a stride of 4.

[0088] In some embodiments, Figure 4Layer1, Layer2, Layer3, and Layer4 in the above example include 2, 2, 6, and 4 residual blocks, respectively, and the number of output channels is 32, 64, 128, and 256, respectively. Each residual block includes: convolution layer, batch normalization layer, and ReLU activation layer. The convolution layer uses the convolutional neural network basic block CBR, the convolution kernel size is 3×3, and the step size and padding can be adaptively set according to actual needs.

[0089] In some embodiments, Figure 4 Each output head (hm, wh, and reg) in the output layer heads consists of one or two convolutional layers. If the structural parameter head_conv configuring the output head is greater than 0, a depth-wise separable convolution followed by a 1×1 convolution can be used; otherwise, a 1×1 convolution can be used directly.

[0090] In some embodiments, the residual block layer may further include at least one deconvolution layer deconv_layers, each of which may be created based on the _make_deconv_layer function. Each deconvolution layer may adopt a convolutional neural network basic block CBR_DW, which adds a depthwise separable convolution (Depthwise Convolution) on the basis of the convolutional neural network basic block CBR. The deconvolution layer (ConvTranspose2d) is used for upsampling.

[0091] It should be noted that depthwise separable convolution is a special convolution method that divides the convolution operation into two steps. First, the convolution operation is performed independently on each channel of the input (i.e., depthwise convolution), and then the outputs of these channels are combined using point convolution (also known as 1×1 convolution). This method can significantly reduce the number of parameters and the amount of computation while maintaining good performance.

[0092] For example, Figure 6 The second structural diagram of a convolutional neural network basic block provided in the embodiment of the present application is as follows: Figure 6 As shown in the figure, this structure is the structure of the convolutional neural network basic block CBR_DW, which includes depth_conv deep convolution layer, batch normalization layer, ReLU activation layer, 1×1 convolution layer, batch normalization layer and ReLU activation layer in sequence.

[0093] It should be noted that if Figure 4The network structure shown is the backbone network of the CenterNet model. The input image (target image) first passes through the convolution layer conv1, and then passes through Layer1, Layer2, Layer3 and Layer4 layer by layer. Each layer contains multiple residual blocks, which gradually reduce the spatial dimension of the feature map and increase the number of feature channels. Then, the feature map is gradually upsampled through the deconvolution layer and fused with the feature map of the previous layer to restore the spatial dimension of the feature map. Finally, each output head processes the upsampled feature map to generate the output of a specific task.

[0094] It is understandable that the backbone network of the CenterNet model provided in the embodiment of the present application is a lightweight network structure, which integrates the improved ResNet with the feature pyramid network to improve the multi-scale target detection performance. In addition, the use of deep separable convolution to reduce parameters and computational complexity, the problem of gradient vanishing is solved through residual connection, and upsampling is used to achieve effective feature fusion. These designs can make the CenterNet model suitable for efficient target detection on resource-constrained embedded devices.

[0095] In some embodiments, the backbone network based on the CenterNet model extracts features from the target image to obtain a first feature map, including:

[0096] Extracting features of the target image based on the target residual network to obtain multiple feature maps of different scales;

[0097] The feature maps of different scales are fused based on the feature pyramid network to obtain the first feature map.

[0098] In an embodiment of the present application, the target image can be first input into the target residual network to use the target residual network to extract features of the target image to obtain feature maps of multiple scales, and then the feature maps of multiple scales are fused based on the feature pyramid network to obtain a first feature map.

[0099] It should be noted that the number of feature maps of multiple scales is determined by the architecture of the target residual network, the size of the target image, the convolutional layer, the pooling layer and other parameters.

[0100] It can be understood that the embodiment of the present application processes the target image jointly through the target residual network and the feature pyramid network, which can simultaneously capture the detail information and global information in the target image, so that the obtained first feature map has a multi-scale feature representation, thereby improving the richness and diversity of the features.

[0101] In some embodiments, fusing the plurality of feature maps of different scales based on the feature pyramid network to obtain the first feature map includes:

[0102] Based on the feature pyramid network, the plurality of feature maps of different scales are merged to obtain a second feature map;

[0103] The second feature map is upsampled based on the deconvolution layer in the CenterNet model to obtain the first feature map.

[0104] In an embodiment of the present application, after a plurality of feature maps of different scales are fused based on a feature pyramid network, the fused second feature map can be upsampled based on a deconvolution layer in a CenterNet model to obtain a first feature map.

[0105] It should be noted that in a convolutional neural network, key features in an image can be extracted through operations such as convolution and pooling. However, these operations can also result in reduced image resolution and loss of details. The deconvolution layer can restore the detailed information of the image to a certain extent through upsampling operations, making the generated image closer to the original image. Therefore, after the embodiment of the present application performs upsampling processing on the fused second feature map through the deconvolution layer, the first feature map obtained can have a higher resolution and restore the detailed information of the original target image.

[0106] For example, Figure 7 A second flow chart of a target detection method provided in an embodiment of the present application is as follows: Figure 7 As shown, the method includes:

[0107] S701, extracting features of the target image based on the target residual network in the backbone network of the CenterNet model to obtain feature maps of multiple different scales;

[0108] S702, fusing the feature maps of different scales using a feature pyramid network in a backbone network based on a CenterNet model to obtain a second feature map;

[0109] S703, performing upsampling processing on the second feature map based on the deconvolution layer in the CenterNet model to obtain a first feature map;

[0110] S704: generating a heat map capable of characterizing the distribution of center points of the target object included in the target image based on the first feature map and the elliptical Gaussian kernel;

[0111] S705: Determine a detection result of the target object based on the heat map.

[0112] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0113] In some embodiments, determining the detection result of the target object based on the heat map includes:

[0114] Based on the first feature map, determining the size of the target object and the offset of the center point coordinates of the target object respectively;

[0115] A detection result of the target object is determined based on the heat map, the size of the target object and the offset of the center point coordinates of the target object.

[0116] In an embodiment of the present application, after obtaining the first feature map, the size prediction branch and the center point offset prediction branch of the CenterNet model can be regressed to obtain the size and center point offset of the target object respectively. Then, in the inference stage, the predicted center point offset can be added to the coordinates of the peak point of the heat map to obtain a more accurate target object center point position. Finally, the predicted target object center point position and the target object size are used as the detection result of the target object.

[0117] For example, Figure 8 A third flow chart of a target detection method provided in an embodiment of the present application is as follows: Figure 8 As shown, the method includes:

[0118] S801, extracting features of the target image based on the target residual network in the backbone network of the CenterNet model to obtain feature maps of multiple scales;

[0119] S802, fusing the feature maps of different scales based on a feature pyramid network in a backbone network of a CenterNet model to obtain a second feature map;

[0120] S803, performing upsampling processing on the second feature map based on the deconvolution layer in the CenterNet model to obtain a first feature map;

[0121] S804: generating a heat map capable of characterizing the distribution of center points of the target object included in the target image based on the first feature map and the elliptical Gaussian kernel;

[0122] S805: Based on the first feature map, determine the size of the target object and the offset of the center point coordinates of the target object respectively;

[0123] S806: Determine a detection result of the target object based on the heat map, the size of the target object, and the offset of the center point coordinates of the target object.

[0124] It should be noted that there is no sequence between S804 and S805 and they can be executed simultaneously, or S804 can be executed first and then S805, or S805 can be executed first and then S804.

[0125] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0126] The following describes a quantitative test of the target detection method proposed in the embodiment of the present application.

[0127] It should be noted that, in view of the long-tail problem that is common in data sets, the traditional mean average precision (mAP) evaluation index gives equal importance to all categories when calculating, which cannot accurately reflect the needs in practical applications. The embodiment of the present application proposes the mAP_ratio indicator, which introduces an attention mechanism and weights the calculation of mAP by considering the proportion of the number of detections in each category. The calculation of mAP_ratio is shown in formula (3):

[0128]

[0129] Among them, n represents the total number of all categories, i represents the current category, and mAP i Indicates the mAP value of the current category, ratio i Indicates the ratio of the number of detections of the current category to the number of detections of all categories. i The sum is equal to 1.

[0130] The experimental results of target detection are shown in Tables 1 and 2. It can be seen from Tables 1 and 2 that the performance of the CenterNet model using the elliptical Gaussian kernel is better than that of the CenterNet model using the circular Gaussian kernel. Moreover, the performance of the improved CenterNet model is better than that of YOLO, and features can be learned quickly on less training data.

[0131] Table 1

[0132] Model Name mAP(%) mAP_ratio(%) CenterNet (circular Gaussian kernel) 39.81 48.23 CenterNet (Elliptical Gaussian Kernel) 46.09 53.03

[0133] Table 2

[0134] Model Name mAP(%) mAP_ratio(%) CenterNet 42.49 50.06 YOLO 40.75 45.50

[0135] It should be noted that the target detection method provided in the embodiment of the present application achieves an effective balance between model size and performance by reducing the size of the CenterNet model based on an improved Gaussian kernel and multi-scale feature fusion technology while improving detection accuracy, efficiency, stability, consistency, adaptability and generalization ability. It is particularly suitable for target detection scenarios that require high precision and real-time performance.

[0136] The target detection device provided in the embodiment of the present application is described below. The target detection device described below and the target detection method described above can be referenced to each other.

[0137] Fig. 9 A schematic diagram of a target detection device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device includes: a feature extraction module 910, a generation module 920 and a determination module 930; wherein:

[0138] A feature extraction module 910 is used to extract features of a target image based on a backbone network of a CenterNet model to obtain a first feature map;

[0139] A generating module 920, configured to generate a heat map capable of representing the distribution of center points of the target object included in the target image based on the first feature map and the elliptical Gaussian kernel;

[0140] A determination module 930, configured to determine a detection result of the target object based on the thermal map;

[0141] Among them, the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object, and there is a specific relationship between the size of the circumscribed rectangle and the intersection-and-union ratio, and the intersection-and-union ratio is the ratio of the intersection area to the union area of ​​the predicted detection box and the real detection box of the target object.

[0142] The target detection device provided in the embodiment of the present application first extracts features of the target image based on the backbone network of the CenterNet model to obtain a first feature map, and then generates a heat map capable of characterizing the distribution of the center points of the target object included in the target image based on the first feature map and the elliptical Gaussian kernel, and then determines the detection result of the target object based on the heat map. In this way, the embodiment of the present application generates a heat map based on the elliptical Gaussian kernel, and the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object, and there is a specific relationship between the size of the circumscribed rectangle and the intersection-and-union ratio, that is, based on the specific relationship, the circumscribed rectangle of the center point distribution of the target object can be determined, and then the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined by the circumscribed rectangle, so that the elliptical Gaussian kernel can accurately reflect the actual shape of the target object, especially for target objects with large differences in aspect ratio, with higher detection accuracy and adaptability, meeting diverse detection needs.

[0143] In some embodiments, the specific relationship between the size of the bounding rectangle and the intersection-over-union ratio is:

[0144]

[0145] Wherein, w represents the width of the circumscribed rectangle, h represents the length of the circumscribed rectangle, iou represents the intersection-and-union ratio, and W TP Represents the width of the real detection box, H TP Represents the length of the real detection box.

[0146] In some embodiments, the backbone network of the CenterNet model is composed of a target residual network and a feature pyramid network.

[0147] In some embodiments, the target residual network includes a convolutional layer and at least one residual block layer, each of the residual block layers includes at least one residual block, and each of the residual blocks includes at least one convolutional layer.

[0148] In some embodiments, the feature extraction module 910 includes:

[0149] A feature extraction unit, used to extract features of the target image based on the target residual network to obtain a plurality of feature maps of different scales;

[0150] A fusion unit is used to fuse the multiple feature maps of different scales based on the feature pyramid network to obtain the first feature map.

[0151] In some embodiments, the fusion unit is further configured to:

[0152] Based on the feature pyramid network, the multiple feature maps of different scales are fused to obtain a second feature map; based on the deconvolution layer in the CenterNet model, the second feature map is upsampled to obtain the first feature map.

[0153] In some embodiments, the determining module 930 includes:

[0154] A first determining unit, configured to determine the size of the target object and the offset of the coordinates of the center point of the target object based on the first feature map;

[0155] The second determination unit is used to determine the detection result of the target object based on the heat map, the size of the target object and the offset of the center point coordinates of the target object.

[0156] It should be noted here that the above-mentioned target detection device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned target detection method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0157] Fig.10 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may execute executable data instructions stored in the memory 1030 to implement some or all of the steps in the target detection method provided in the above embodiments.

[0158] In addition, the executable data instructions stored in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0159] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, some or all of the steps in the target detection method provided in the above embodiments are implemented.

[0160] An embodiment of the present application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, some or all of the steps in the target detection method provided in the above embodiments are implemented.

[0161] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] The above description is merely an optional embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A target detection method, characterized in that: include: Based on the backbone network of the CenterNet model, features of the target image are extracted to obtain the first feature map; Based on the first feature map and the elliptical Gaussian kernel, generating a heat map capable of characterizing the distribution of center points of the target object included in the target image; Based on the thermal map, determining a detection result of the target object; Among them, the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object, and there is a specific relationship between the size of the circumscribed rectangle and the intersection-and-union ratio, and the intersection-and-union ratio is the ratio of the intersection area to the union area of ​​the predicted detection box and the real detection box of the target object.

2. The target detection method according to claim 1, characterized in that: The specific relationship between the size of the bounding rectangle and the intersection-over-union ratio is: Wherein, w represents the width of the circumscribed rectangle, h represents the length of the circumscribed rectangle, iou represents the intersection-and-union ratio, and W TP Represents the width of the real detection box, H TP Represents the length of the real detection box.

3. The target detection method according to claim 1 or 2, characterized in that: The backbone network of the CenterNet model is composed of a target residual network and a feature pyramid network.

4. The target detection method according to claim 3, characterized in that: The target residual network includes a convolutional layer and at least one residual block layer, each of the residual block layers includes at least one residual block, and each of the residual blocks includes at least one convolutional layer.

5. The target detection method according to claim 3, characterized in that: The backbone network based on the CenterNet model performs feature extraction on the target image to obtain a first feature map, including: Extracting features of the target image based on the target residual network to obtain multiple feature maps of different scales; The feature maps of different scales are fused based on the feature pyramid network to obtain the first feature map.

6. The target detection method according to claim 5, characterized in that: The fusing the plurality of feature maps of different scales based on the feature pyramid network to obtain the first feature map includes: Based on the feature pyramid network, the plurality of feature maps of different scales are merged to obtain a second feature map; The second feature map is upsampled based on the deconvolution layer in the CenterNet model to obtain the first feature map.

7. The target detection method according to claim 1 or 2, characterized in that: The step of determining a detection result of the target object based on the heat map includes: Based on the first feature map, determining the size of the target object and the offset of the center point coordinates of the target object respectively; A detection result of the target object is determined based on the heat map, the size of the target object and the offset of the center point coordinates of the target object.

8. A target detection device, characterized in that: include: A feature extraction module is used to extract features of the target image based on the backbone network of the CenterNet model to obtain a first feature map; A generating module, configured to generate a heat map capable of characterizing the distribution of center points of a target object included in the target image based on the first feature map and an elliptical Gaussian kernel; A determination module, used to determine a detection result of the target object based on the thermal map; Among them, the major axis kernel radius and the minor axis kernel radius of the elliptical Gaussian kernel are determined based on the circumscribed rectangle of the center point distribution of the target object, and there is a specific relationship between the size of the circumscribed rectangle and the intersection-and-union ratio, and the intersection-and-union ratio is the ratio of the intersection area to the union area of ​​the predicted detection box and the real detection box of the target object.

9. An electronic device, characterized in that: include: A memory for storing executable data instructions; A processor, configured to implement the steps of the target detection method according to any one of claims 1 to 7 when executing the executable data instructions stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target detection method according to any one of claims 1 to 7 are implemented.

Images