Target detection methods and equipment

Abstract

This disclosure provides a target detection method and apparatus. The method includes: acquiring an image to be detected; detecting targets in the image using a first model to obtain an attention map and a density map of the image; and generating a target detection map of the image based on the attention map and the density map, wherein the target detection map includes location markers of the targets in the image. The first model is a lightweight deep learning model trained based on sample data, a second model, and a first loss function. The first loss function includes a distillation loss function that reflects the distillation loss between the first and second models, and the model size of the second model is larger than that of the first model. Therefore, by using a lightweight deep learning model trained based on knowledge distillation, the efficiency of target detection in the image is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a target detection method and device. Background Technology

[0002] As the business grows, there is a need to detect targets in complex scenarios, such as pedestrians in street scenes and vehicles in road scenes.

[0003] Currently, density map estimation-based methods are used to achieve target detection in complex scenes, reflecting not only the number of targets but also their distribution within the scene. However, this approach requires a trade-off between superior target detection performance and real-time performance. Typically, images are acquired by a camera and then transmitted to a backend server, where the server's powerful graphics processor performs target detection, resulting in poor real-time performance.

[0004] Therefore, the efficiency of target detection needs to be improved. Summary of the Invention

[0005] This disclosure provides a target detection method and apparatus to improve the efficiency of target detection.

[0006] In a first aspect, embodiments of this disclosure provide a target detection method, including:

[0007] Acquire the image to be detected;

[0008] The first model is used to detect targets in the image, and the attention map and density map of the image are obtained.

[0009] Based on the attention map and the density map, a target detection map of the image is generated, wherein the target detection map includes the location identifier of the target in the image;

[0010] The first model is a lightweight deep learning model trained based on sample data, a second model, and a first loss function. The first loss function includes a distillation loss function that reflects the distillation loss between the first model and the second model. The model size of the second model is larger than that of the first model.

[0011] Secondly, embodiments of this disclosure provide a target detection device, comprising:

[0012] The acquisition unit is used to acquire the image to be detected;

[0013] A detection unit is used to detect targets in the image using a first model, and to obtain an interest map and a density map of the image.

[0014] A generation unit is configured to generate a target detection map of the image based on the attention map and the density map;

[0015] The first model is a lightweight deep learning model trained based on sample data, a second model, and a first loss function. The first loss function includes a distillation loss function that reflects the distillation loss between the first model and the second model. The model size of the second model is larger than that of the first model.

[0016] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory;

[0017] The memory stores computer-executed instructions;

[0018] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the target detection method as described in the first aspect and various possible designs of the first aspect.

[0019] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the target detection method described in the first aspect and various possible designs of the first aspect.

[0020] Fifthly, embodiments of this disclosure provide a computer program product:

[0021] The target detection method and apparatus provided in this embodiment detect targets in an image using a first model, obtaining an interest map and a density map of the image. Based on the interest map and density map, a target detection map of the image is generated. The first model is a lightweight deep learning model trained based on sample data, a second model, and a first loss function. The first loss function includes a distillation loss function that reflects the distillation loss between the first and second models. The second model has a larger model size than the first model. Therefore, by distilling the larger second model, a lightweight first model is obtained. Target detection is then performed on the image based on this lightweight first model, improving the target detection efficiency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 An example diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0024] Figure 2 This is a schematic diagram of the target detection method provided in the embodiments of this disclosure;

[0025] Figure 3 A flowchart illustrating a single training process of the first model in the target detection method provided in this embodiment of the disclosure;

[0026] Figure 4 A flowchart illustrating a single training process of the second model in the target detection method provided in this embodiment of the disclosure;

[0027] Figure 5 Example diagrams of the model structures of the first and second models provided in the embodiments of this disclosure;

[0028] Figure 6 This is a structural block diagram of the target detection device provided in the embodiments of this disclosure;

[0029] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0031] refer to Figure 1 , Figure 1 This is an example diagram illustrating an application scenario provided by an embodiment of this disclosure.

[0032] like Figure 1 As shown, the application scenario is, for example, a road scene, and the target is, for example, pedestrians. In the road scene, video is captured by a roadside camera 101, and the camera 101 or a terminal 102 connected to the camera 101 performs real-time pedestrian detection on the images in the video to obtain the number of pedestrians in the images.

[0033] Among these, the camera and the terminal connected to the camera are lightweight devices with limited computing power. How to achieve real-time target detection on these lightweight devices is one of the most pressing problems to be solved.

[0034] Detecting pedestrians in images using real-time crowd counting models typically involves the following two methods:

[0035] One approach is based on pedestrian detection. This type of method uses traditional Histogram of Oriented Gradient (HOG) detectors or deep learning algorithms to detect pedestrians in images. Examples of deep learning algorithms include YOLO and Region Convolutional Neural Networks (RCNNs). However, this approach performs poorly in situations with significant pedestrian occlusion, and is prone to false positives and false negatives.

[0036] Method 2: Head Density Map Estimation. This method detects heads in images, providing a density map reflecting the number and distribution of pedestrians. However, this method requires a trade-off between accuracy and real-time performance in head detection. The typical approach involves sending images captured by a camera to a backend server, utilizing the server's powerful image processing capabilities for head detection. This involves using a computationally intensive model trained with traditional binary cross-entropy loss, resulting in both efficiency and accuracy requiring improvement.

[0037] To address the aforementioned problems, this disclosure provides an object detection method. In this method, a smaller, more computationally intensive first model is obtained by utilizing a second model with a larger model size and lower computational cost, along with knowledge distillation. This allows the first model to be easily deployed on lightweight devices. During object detection, the first model is used to detect objects in an image, resulting in an interest map and a density map. Based on the interest map and density map, an object detection map is generated. Therefore, on the one hand, the lightweight first model improves the efficiency of object detection, facilitating real-time object detection on lightweight devices; on the other hand, the obtained object detection map reflects both the quantity and distribution of objects, ensuring the accuracy of object detection.

[0038] Optionally, the target is pedestrians, and the target detection is pedestrian detection and head detection in the image.

[0039] For example, the target detection method provided in this disclosure can be applied to a terminal or a server. When applied to a terminal, the method enables real-time target detection of images acquired by the terminal. When applied to a server, the method enables target detection of images sent by the terminal. The terminal device can be a personal digital assistant (PDA) device, a handheld device with wireless communication capabilities (e.g., a smartphone, tablet), a computing device (e.g., a personal computer, PC), an in-vehicle device, a wearable device (e.g., a smartwatch, smart bracelet), or a smart home device (e.g., a smart display device), etc.

[0040] Optionally, the terminal is a camera, and the first model is deployed in the camera's chip. This enables real-time target detection of images captured by the camera, achieving real-time target detection from the camera's side.

[0041] refer to Figure 2 , Figure 2 This is a schematic flowchart of the target detection method provided in an embodiment of this disclosure. Figure 2 As shown, the target detection method includes:

[0042] S201: Acquire the image to be detected.

[0043] In one example, the image to be detected can be an image captured in real time by a camera, or one or more frames of an image obtained from a video captured in real time by a camera.

[0044] In another example, the image to be detected can be an image input by the user or an image selected by the user. For example, the user inputs or selects the image to be detected on the terminal's display interface. Alternatively, the server receives the user input or the image selected by the user from the terminal.

[0045] In another example, the image to be detected can be an image playing in real time on the terminal. For instance, when the terminal detects an image or video playing, it acquires the currently playing image or video frame. This enables target detection of an image playing in real time on the terminal.

[0046] S202: Detect targets in the image using the first model to obtain the attention map and density map of the image.

[0047] The first model is a lightweight deep learning model trained based on sample data, the second model, and the first loss function. The first loss function includes a distillation loss function that reflects the distillation loss between the first model and the second model. The second model is a deep learning model, and the model size of the second model is larger than that of the first model.

[0048] During the training of the first model, a second model is acquired. Based on the sample images in the sample data and the distillation loss function in the first loss function, knowledge distillation is performed on the second model and the first model. Thus, the training of the first model is guided by the larger-scale, more accurate second model, thereby improving the target detection accuracy of the lightweight first model.

[0049] In this step, after obtaining the image to be detected, the image can be input into the first model, or the image can be preprocessed (e.g., image denoising, cropping, enhancement, etc.) and then input into the first model. Multiple convolutional layers in the first model perform feature extraction and decoding on the image, extracting the attention map and density map respectively. The attention map focuses on the clarity and accuracy of target detection in the image, more clearly reflecting the position of a single target in the image, while the density map focuses on the comprehensiveness of target detection in the image, more comprehensively reflecting the distribution of multiple targets in the image.

[0050] S203: Generate the object detection map of the image based on the attention map and density map.

[0051] The target detection map includes location markers for targets within the image. For example, when the target is a pedestrian, highlighted points can be used as location markers for the target in the image. Thus, the number and distribution of multiple highlighted points in the image reflect the number and distribution of multiple targets in the image.

[0052] In this step, after obtaining the attention map and density map, the attention map and density map can be combined to obtain the attention-bearing density map, i.e., the target detection map. This not only reflects the density distribution of multiple targets in the image, but also improves the accuracy of the target location reflected by the target detection map, thereby improving the clarity, accuracy and comprehensiveness of target detection in the image and effectively improving the target detection effect.

[0053] The combination of the attention map and the density map is the combination of each pixel in the attention map and each pixel in the density map, which is to merge each pixel in the attention map into each pixel in the density map.

[0054] In this embodiment of the disclosure, a lightweight first model is obtained based on sample data, a second model, and knowledge distillation. The first model is used to detect targets in the image, resulting in an interest map and a density map of the image. By combining the interest map and the density map, a target detection map reflecting the position of the target in the image is obtained. Thus, the efficiency of target detection is improved by using a lightweight first model that can be deployed on lightweight devices, and the accuracy of target detection is improved by combining the interest map and the density map through knowledge distillation.

[0055] Regarding the model structures of the first and second models:

[0056] In some embodiments, the first model includes at least two network branches, each including one or more convolutional layers. The at least two network branches include an attention map branch and a density map branch. During training of the first model, the attention map branch is used to learn the attention map of the target in the sample image, and the density map branch is used to learn the density map of the target in the sample image. When the first model performs target detection on the image, the attention map branch is used to extract the attention map of the image, and the density map branch is used to extract the density map of the image. For ease of description, the attention map branch in the first model is referred to as the first attention map branch, and the density map branch in the first model is referred to as the first density map branch.

[0057] Specifically, the first model includes multiple feature extraction layers, multiple feature decoding layers, a first attention map branch, and a first density map branch. In the first model, the last feature extraction layer is connected to the first feature decoding layer, and the last feature decoding layer is connected to the first convolutional layer in both the first attention map branch and the first convolutional layer in the first density map branch.

[0058] The first model also includes a network layer for fusing the attention map and the density map. This network layer is connected to the last convolutional layer in the first attention map branch and the last convolutional layer in the first density map branch, respectively. It receives the attention map output by the first attention map branch and the density map output by the first density map branch, fuses the attention map and the density map to obtain the target detection map and outputs it.

[0059] In some embodiments, the model structure of the second model is the same as or similar to that of the first model. The second model includes at least two network branches, each including one or more convolutional layers. These at least two network branches include an attention map branch and a density map branch. The details of the attention map branch and density map branch can be referenced from those in the first model and will not be repeated here. For ease of description, the attention map branch in the second model is referred to as the second attention map branch, and the density map branch in the second model is referred to as the second density map branch.

[0060] Specifically, the second model includes multiple feature extraction layers, multiple feature decoding layers, a second attention map branch, and a second density map branch. In the second model, the last feature extraction layer is connected to the first feature decoding layer, and the last feature decoding layer is connected to the first convolutional layer in both the first attention map branch and the first convolutional layer in the first density map branch.

[0061] The second model also includes a network layer for fusing the attention map and the density map. This network layer is connected to the last convolutional layer in the second attention map branch and the last convolutional layer in the second density map branch, respectively. It receives the attention map output by the second attention map branch and the density map output by the second density map branch, fuses the attention map and the density map to obtain the target detection map and outputs it.

[0062] Optionally, in the first model and / or the second model, the feature extraction layer is the Ghost module in the GhostNet edge neural network architecture. Thus, the Ghost module is used in the first model and / or the second model to replace the convolutional layer for feature extraction. Taking advantage of the Ghost module's advantages of extracting more features with fewer parameters and lower computational cost, the object detection efficiency and object detection accuracy of the first model are improved.

[0063] Optionally, in the first model and / or the second model, the feature extraction and decoding process employs U-Net mode for information augmentation to improve the accuracy of the feature map obtained after feature decoding, thereby improving the target detection accuracy of the first model and / or the second model.

[0064] For example, in the U-Net mode of the first model, the feature map output by the first feature extraction layer is merged with the feature map output by the second-to-last feature decoding layer to obtain the input features of the second-to-last feature decoding layer; the feature map output by the second feature extraction layer is merged with the feature map output by the third-to-last feature decoding layer to obtain the input features of the second-to-last feature decoding layer, and so on.

[0065] Regarding the model sizes of the first and second models:

[0066] In some embodiments, the second model has more network layers than the first model. Therefore, by using more network layers in the second model, the target detection accuracy of the second model is improved, thereby improving the target detection accuracy of the first model obtained based on the second model and knowledge distillation.

[0067] For example, the number of feature extraction layers in the second model is greater than the number of feature extraction layers in the first model, or the number of feature decoding layers in the second model is greater than the number of feature decoding layers in the first model.

[0068] In some embodiments, the number of network layers in the second model is equal to the number of network layers in the first model. In this case, the number of channels in the feature extraction layer of the second model is greater than the number of channels in the corresponding feature extraction layer of the first model, and / or, the number of channels in the feature decoding layer of the second model is greater than the number of channels in the corresponding feature decoding layer of the first model. Therefore, in the feature extraction layer and / or feature decoding layer of the second model, a larger model capacity is used to capture image features that the first model may not easily capture, thereby improving the feature extraction accuracy of the first model trained based on the second model and knowledge distillation, and thus improving the target detection accuracy of the first model.

[0069] As an example, the number of channels in the feature extraction layer of the second model being greater than the corresponding number of channels in the feature extraction layer of the first model means that the number of channels in the first, second, ..., Kth feature extraction layers of the second model is greater than the number of channels in the first, second, ..., Kth feature extraction layers of the first model, respectively. Here, the number of channels in the feature extraction layer includes both the number of input channels and the number of output channels.

[0070] Furthermore, the number of channels in the feature extraction layer of the second model is a preset multiple of the number of channels in the corresponding feature extraction layer of the first model, and / or the number of channels in the feature decoding layer of the second model is a preset multiple of the number of channels in the corresponding feature decoding layer of the first model, wherein the preset multiple is greater than 1. Thus, the relationship between the size of the feature extraction layer and / or feature decoding layer in the second model and the size of the feature extraction layer and / or feature decoding layer in the first model is established more stably through this multiple relationship.

[0071] In some embodiments, based on the model structure of the first model described above, a possible implementation of S202 includes: in the feature extraction layer of the first model, performing feature extraction on the image through convolution and downsampling operations; in the feature decoding layer of the first model, performing feature decoding on the extracted features through convolution and upsampling operations to obtain image features (also known as feature activation values); inputting the image features into the first attention map branch and the first density map branch respectively to obtain the attention map and density map of the image, thereby realizing the extraction of the attention map and density map of the image in the first model.

[0072] In some embodiments, a possible implementation of S203 is: performing a dot product operation on the attention map and the density map to obtain a density map with attention. Thus, by using the dot product operation, the pixels in the attention map are combined with the pixels in the density map, improving the combination effect of the attention map and the density map. Specifically, performing a dot product operation on the attention map and the density map means treating the attention map as a first matrix composed of multiple pixel values, and the density map as a second matrix composed of multiple pixel values, performing a dot product operation on the first matrix and the second matrix, and the result is the density map with attention.

[0073] Furthermore, the attention map is processed using an activation function, and the processed attention map and density map are then multiplied by a dot product to obtain a density map with attention. This improves the fusion effect of the attention map within the density map and enhances the accuracy of the target detection map in reflecting the quantity and density distribution of targets in the image. For example, the sigmoid function can be used as the activation function.

[0074] The training process of the first model and the training process of the second model will be described in subsequent examples.

[0075] The following points need to be clarified: 1. The training process of the first model and the training process of the second model are conducted separately; 2. The training processes of the first model and the second model can be performed on the same device or on different devices; 3. The application process of the first model in image object detection and the training process of the first model can be performed on the same device or on different devices. For example, the trained first model can be deployed on a lightweight device for image object detection; the first model can be trained on a lightweight device, or on a computer, server, or other similar device, while the training process of the second model can be performed on a computer, server, or other similar device.

[0076] refer to Figure 3 , Figure 3 This is a flowchart illustrating a single training process of the first model in the target detection method provided in this embodiment of the disclosure. Figure 3 As shown, the training process of the first model includes:

[0077] S301: Extract and decode features from the sample image using the first model.

[0078] As described above regarding the network structure of the first model, the first model includes multiple network layers, including an input layer, intermediate layers, and an output layer. In one example, the first model includes an input layer, multiple feature extraction layers, multiple feature decoding layers, multiple convolutional layers in a first attention map branch, multiple convolutional layers in a first density map branch, and an output layer that fuses the attention map and the density map to output a target detection map. The multiple feature extraction layers, multiple feature decoding layers, multiple convolutional layers in the first attention map branch, and multiple convolutional layers in the first density map branch are intermediate layers.

[0079] In this step, the first model extracts and decodes the features of the sample images in the sample data, which can obtain the image features output by each intermediate layer and the feature map output by the output layer in the first model. This feature map is the target detection map obtained by the first model to perform target detection on the sample images. For easy distinction, the feature map of the sample images output by the output layer in the first model is called the first feature map.

[0080] S302: The sample image is extracted and decoded using the second model.

[0081] As described above regarding the network structure of the second model, it comprises multiple network layers, including an input layer, intermediate layers, and an output layer. In one example, the second model includes an input layer, multiple feature extraction layers, multiple feature decoding layers, multiple convolutional layers in the second attention map branch, multiple convolutional layers in the second density map branch, and an output layer that fuses the attention map and density map to output a target detection map. The multiple feature extraction layers, multiple feature decoding layers, multiple convolutional layers in the second attention map branch, and multiple convolutional layers in the second density map branch constitute intermediate layers.

[0082] In this step, the second model extracts and decodes features from the sample images in the sample data, yielding image features output by each intermediate layer and feature maps output by the output layer. These feature maps are the target detection maps obtained by the second model when performing target detection on the sample images. For ease of distinction, the feature map of the sample images output by the output layer of the second model is referred to as the second feature map.

[0083] S303: Determine the value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model.

[0084] In this step, the difference between the image features extracted by the first model and the image features extracted by the second model is determined. Based on this difference, the function value of the distillation loss function in the first loss function is determined. Then, based on the function value of the distillation loss function, the function value of the first loss function is determined. If the first loss function only includes the distillation loss function, then the function value of the distillation loss function is the function value of the first loss function.

[0085] S304: Correct the first model based on the loss value of the first loss function.

[0086] In this step, after determining the value of the first loss function, the model parameters of the first model are adjusted based on the value of the first loss function and the optimization algorithm (e.g., gradient descent), thus correcting the first model. Therefore, during the correction of the first model, the distillation loss function in the first loss function guides the image features extracted by the first model to approximate the image features extracted by the second model, making the object detection accuracy of the first model approach that of the second model. After correcting the first model, one training cycle of the first model is completed. At this point, it can be determined whether the value of the first loss function is less than a first threshold. If so, the trained first model is obtained; otherwise, the next training cycle of the first model continues until the value of the first loss function is less than the first threshold.

[0087] In some embodiments, the distillation loss function includes a structural loss function. In this case, one possible implementation of S303 includes: determining the value of the structural similarity loss function based on the difference between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model. Thus, based on the difference between the first and second feature maps reflected by the structural similarity loss function, the training of the first model is guided, causing the first model to optimize in a direction where the first feature map output by the first model approximates the second feature map output by the second model.

[0088] Optionally, the structural similarity between the first and second feature maps is first determined. Structural similarity includes at least one of the following: brightness similarity, contrast similarity, and pixel vector similarity. Then, based on the structural similarity between the first and second feature maps, the function value of the structural similarity loss function is determined. The similarity between the first and second feature maps in one or more aspects such as brightness, contrast, and pixel vectors reflects the degree of difference between them. For example, the greater the brightness similarity between the first and second feature maps, the smaller the difference in brightness. Therefore, by combining the similarity between the first and second feature maps in one or more aspects such as brightness, contrast, and pixel vectors to obtain the function value of the structural similarity loss function, the accuracy of the function value of the structural similarity loss function is improved.

[0089] Specifically, the first feature map and the second feature map can be compared in one or more aspects, including brightness, contrast, and pixel vectors, to obtain one or more comparison results in terms of brightness similarity, contrast similarity, and pixel vector similarity between the first and second feature maps. Combining these one or more comparison results, the structural similarity between the first and second feature maps is obtained.

[0090] Optionally, the structural similarity between the first feature map and the second feature map can be determined by the Structural Similarity (SSIM) calculation method. By utilizing SSIM, a perception capability closer to that of the Human Visual System (HVS) can be provided, thereby improving the accuracy of the calculation of the structural similarity between the first feature map and the second feature map.

[0091] In the SSIM method, the first and second feature maps are normalized separately. The similarity between the normalized first and second feature maps in terms of photometric quality, contrast, and pixel vectors is calculated to obtain the brightness similarity, contrast similarity, and pixel vector similarity between the first and second feature maps. Finally, the brightness similarity, contrast similarity, and pixel vector similarity between the first and second feature maps are multiplied to obtain the structural similarity between them.

[0092] Furthermore, the formulas for calculating the brightness similarity, contrast similarity, and pixel vector similarity between the first feature map and the second feature map using SSIM can be as follows:

[0093]

[0094]

[0095] Where, μa μ b Let σ represent the mean pixel value of the first feature map and the mean pixel value of the second feature map, respectively. a σ b Let c1, c2, and c3 represent the pixel value variances of the first and second feature maps, respectively. c1, c2, and c3 are small constants to avoid division by zero. L(a,b) represents the brightness similarity between the first and second feature maps, C(a,b) represents the contrast similarity between the first and second feature maps, and S(a,b) represents the pixel vector similarity between the first and second feature maps.

[0096] At this point, the formula for calculating the structural similarity between the first feature map and the second feature map can be: SSIM(a,b) = L(a,b) * C(a,b) * S(a,b). Where SSIM(a,b) represents the structural similarity between the first feature map and the second feature map. The structural similarity loss function can be expressed as: L ssim =1-SSIM(a,b). Therefore, after obtaining the structural similarity between the first and second feature maps, the function value of the structural similarity loss function can be calculated.

[0097] Optionally, besides directly determining the structural similarity between the entire first feature map and the entire second feature map, the first and second feature maps can be divided into multiple image regions respectively; the structural similarity between these multiple image regions of the first feature map and the multiple image regions of the second feature map can then be determined. Based on the structural similarity between these multiple image regions of the first and second feature maps, the structural similarity between the first and second feature maps is determined. This image region division method improves the accuracy of the structural similarity determination between the first and second feature maps.

[0098] In this design, the multiple image regions in the first feature image and the multiple image regions in the second feature image are all of the same size. Based on the position of the image regions in the feature image, the multiple image regions in the first feature image correspond one-to-one with the multiple image regions in the second feature image, and two image regions in the first feature image and the second feature image that correspond to each other can form an image region pair.

[0099] For each image region pair formed by multiple image regions of the first feature image and multiple image regions of the second feature image, the structural similarity between the two image regions in the image region pair is determined, i.e., the structural similarity of the image region pair. Based on the structural similarity of each image region pair, the structural similarity between the first feature image and the second feature image is determined. Specifically, the structural similarity between the first feature image and the second feature image can be determined as the average of the structural similarities corresponding to all image region pairs.

[0100] For example, the first feature map and the second feature map are each divided into I image regions of equal size. Structural similarity between corresponding image regions in the first feature map and the second feature map is determined using SSIM, resulting in the structural similarity of I image region pairs. The mean of the structural similarity of the I image region pairs is calculated to obtain the structural similarity between the first and second feature maps. The formula for calculating the structural similarity between the first and second feature maps can be expressed as:

[0101] Among them, a i b represents the i-th image region in the first feature map. i SSIM(a) represents the i-th image region in the second feature map. i ,b i ) represents a i With b i The structural similarity is given by I, which represents the number of image regions in the first feature map, i.e., the number of image regions in the second image map. The structural similarity loss function can then be expressed as: L ssim =1-MSSSIM(a,b).

[0102] In some embodiments, based on the network structure where "the number of network layers in the second model is equal to the number of network layers in the first model, and the number of channels in the feature extraction layer of the second model is a preset multiple of the number of channels in the corresponding feature extraction layer of the first model and / or the number of channels in the feature decoding layer of the second model is a preset multiple of the number of channels in the feature decoding layer of the first model," the intermediate layers of the teacher network contain more channels, enabling the extraction of richer information. The output of the intermediate layers of the teacher model can be used as supervisory information to guide the training of the student model, allowing the student model to obtain more information from the intermediate layers in addition to the output layer. Therefore, the distillation loss function may include a channel loss function, whose function value reflects the difference between the output of the intermediate layers of the first model and the output of the intermediate layers of the second model, thereby guiding the training of the first model based on this difference and enriching the intermediate layers of the first model. In other words, it makes the feature extraction and / or feature decoding capabilities of the intermediate layers of the first model approximate the feature extraction and / or feature decoding capabilities of the intermediate layers of the second model.

[0103] Based on the distillation loss function, which includes a channel loss function, a channel convolutional layer connects the intermediate layers of the first model and the intermediate layers of the second model. This channel convolutional layer establishes a mapping relationship between the image features output by the intermediate layers of the first model and the image features output by the intermediate layers of the second model. In this mapping relationship, since the number of network layers in the second model is equal to the number of network layers in the first model, there is a one-to-one mapping between the network layers of the second model.

[0104] At this point, the intermediate layer of the first model includes the feature extraction layer of the first model and / or the feature extraction layer of the second model, and the intermediate layer of the second model includes the feature extraction layer of the second model and / or the feature extraction layer of the second model. Thus, through the channel convolutional layer, a one-to-one mapping relationship is established between multiple feature extraction layers of the first model and multiple feature extraction layers of the second model, and / or a one-to-one mapping relationship is established between multiple feature decoding layers of the first model and multiple feature decoding layers of the second model.

[0105] At this point, a possible implementation of S303 further includes: determining the function value of the channel distillation loss function based on the difference between the mutually mapped image features in the image features output from the intermediate layer of the first model and the image features output from the intermediate layer of the second model. Thus, based on the difference between the image features output from the intermediate layer of the first model and the image features output from the intermediate layer of the second feature map, as reflected by the channel distillation loss function, the training of the first model is guided, causing the first model to be optimized in a direction that approximates the intermediate layer output of the second model, thereby improving the feature extraction accuracy of the first model.

[0106] Optionally, in determining the function value of the channel distillation loss function based on the differences between mutually mapped image features, the difference between each pair of mutually mapped image features can be calculated using the L2 paradigm. The function value of the channel distillation loss function is then determined to be the sum of the differences between the mutually mapped image features. Further, the channel distillation loss function can be expressed as:

[0107] in, z represents the image features output by the m-th intermediate layer in the second model. m (x) represents the image features output by the m-th intermediate layer in the first model. With z m (x) represents a pair of mutually mapped image features. This indicates that the result is obtained through L2 normal form calculation. With z m The difference between (x) and h m L represents the channel convolutional layer, M represents the number of intermediate layers in the second model (e.g., the number of feature extraction layers, the number of feature decoding layers, and the total number of feature extraction and feature decoding layers), and L represents the number of intermediate layers in the second model. dis This represents the channel distillation loss function.

[0108] Among them, the channel convolutional layer is a learnable convolutional layer, whose parameters can be adjusted during the training of the first model. In this way, the first model and the channel convolutional layer are optimized by means of the mean square error between the mutually mapped image features, so as to minimize the function value of the channel distillation loss function.

[0109] Furthermore, the channel convolutional layer is a 1*1 learnable convolutional layer. Through the 1*1 learnable convolutional layer, the output of the intermediate layer of the first model is mapped to the output of the corresponding intermediate layer of the second model, thereby increasing the dimensionality of the channels of the intermediate layer of the first model to be consistent with the channels of the corresponding intermediate layer of the second model.

[0110] In some embodiments, based on the model structure of the first model including a feature extraction layer, a feature decoding layer, a first attention map branch, and a first density map branch, the label data corresponding to the sample image includes the actual attention map and the actual density map of the sample image, and the first loss function further includes a first attention map branch loss function and a first density map branch loss function. In this case, a possible implementation of S303 includes: inputting the image features output by the feature decoding layer in the first model into the first attention map branch and the first density map branch respectively to obtain the first attention map and the first density map; determining the function values ​​of the first attention map branch loss function and the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map. Thus, based on the function values ​​of the first attention map branch loss function and the first density map branch loss function, the training process of the first model is guided, improving the accuracy of the first attention map and the first density map output by the first model, so that through continuous training, the first attention map output by the first model gradually approaches the actual attention map of the sample image, and the first density map output by the first model gradually approaches the actual density map of the sample image.

[0111] Optionally, in the process of determining the function values ​​of the first attention map branch loss function and the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map, the function values ​​of the first attention map branch loss function and the first density map branch loss function can be determined based on the differences between the first attention map and the actual attention map. For example, the average pixel difference between the first attention map and the actual attention map can be determined as the function value of the first attention map branch loss function, and the average pixel difference between the first density map and the actual density map can be determined as the function value of the first density map branch loss function.

[0112] Optionally, in the process of determining the function values ​​of the first attention map branch loss function and the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map, the first attention map and the first density map can be fused (the specific fusion method can be referred to the description in the foregoing embodiments) to obtain the first target detection map of the sample image. The function value of the first attention map branch loss function is determined based on the difference between the first attention map and the actual attention map. The function value of the first density map branch loss function is determined based on the difference between the first target detection map and the actual density map. This makes the first attention map of the sample image output by the first model approximate the actual attention map of the sample image, and the first target detection map of the sample image output by the first model approximate the actual density map of the sample image, thereby improving the accuracy of the target detection map output by the first model.

[0113] For example, inputting the image features output from the feature decoding layer in the first model into the first attention map branch yields the first attention map, which can be represented as: A = f(X); inputting the image features output from the feature decoding layer in the first model into the first density map branch yields the first density map, which can be represented as: D = f(X); by performing a dot product operation on the first attention map and the first density map, the first target detection map of the sample image can be obtained, and the calculation formula for the first target detection map can be expressed as: D F =σ(A)⊙D. Where X represents the image features output by the feature decoding layer in the first model, A represents the first attention map of the sample image, and D represents the first density map of the sample image. F This represents the first object detection map of the sample image, ⊙ represents the dot product operation, and σ() represents the activation function, such as the sigmoid function.

[0114] Furthermore, the formula for the first attention map branch loss function can be expressed as:

[0115] Where N represents the number of sample images used in one training cycle of the first model, and A i This represents the first attention map of the i-th sample image obtained after processing by the first model. L represents the actual attention map of the i-th sample image. att Let represent the first attention map branch loss function. Therefore, by averaging the differences between the first attention map and the actual attention map of all sample images during one training cycle of the first model, the function value of the first attention map branch loss function is obtained, thus improving the accuracy of the first attention map branch loss function in reflecting the differences between the first attention map and the actual attention map of the sample images.

[0116] Optionally, the formula for the first density map branch loss function can be expressed as:

[0117] in, This represents the first target detection map of the i-th sample image after processing by the first model. L represents the actual density map of the i-th sample image. den Let represent the first density map branch loss function. Therefore, by averaging the differences between the first object detection map and the actual density map for all samples during one training cycle of the first model, the function value of the first density map branch loss function is obtained, thus improving the accuracy of the first density map branch loss in reflecting the difference between the first object detection map and the actual density map of the sample image.

[0118] Based on the above embodiments, it can be seen that the first loss function may include one or more of the first attention map branch loss function, the first density map branch loss function, and the distillation loss function, and the distillation loss function may include one or more of the structural similarity loss function and the channel distillation loss function.

[0119] When the first loss function includes a first attention map branch loss function, a first density map branch loss function, and a distillation loss function, one possible implementation of S304 includes: weighted summing of the function values ​​of the first attention map branch loss function, the first density map branch loss function, and the distillation loss function to obtain the final loss value of the first model; and correcting the first model based on the final loss value. Thus, by combining the function values ​​of the first attention map branch loss function, the first density map branch loss function, and the distillation loss function obtained based on sample data, the second model, and the first model during training, and correcting the first model based on the final loss value obtained after combination, the training effect of the first model is improved from multiple aspects.

[0120] Optionally, based on distillation loss functions including structural similarity loss function and channel distillation loss function, the overall optimization objective of the first model is L. s It can be represented as:

[0121] L s =w1×L den +w2×L att +w3×L dis +w4×L ssim Where w1, w2, w3, and w4 are preset weights. Furthermore, L s =L den +0.1×L att +L dis +L ssim Based on this optimization objective, the final loss value of the first model can be obtained.

[0122] refer to Figure 4 , Figure 4This is a flowchart illustrating a single training process of the second model in the target detection method provided in this embodiment of the disclosure. Figure 4 As shown, the training process of the second model includes:

[0123] S401. The sample image is extracted and decoded using the second model.

[0124] During the training of the second model, the sample data used can be the same as that used to train the first model, or it can be different from the sample data used to train the first model. The sample data used to train the second model includes sample images and label data for those images.

[0125] In this step, the second model extracts and decodes features from the sample images in the sample data, yielding image features output by each intermediate layer and feature maps output by the output layer. These feature maps are the target detection maps obtained by the second model when performing target detection on the sample images. For ease of distinction, the feature map of the sample images output by the output layer of the second model during training is referred to as the third feature map.

[0126] S402. Determine the value of the second loss function based on the third feature map output from the second model and the label data corresponding to the sample images.

[0127] In this step, after obtaining the third feature map output by the second model, the difference between the third feature map and the label data corresponding to the sample image is determined, and the function value of the second loss function is determined based on the difference between the third feature map and the label data corresponding to the sample image.

[0128] S403. Based on the function value of the second loss function, modify the second model.

[0129] In this step, based on the function value of the second loss function and the optimization algorithm (e.g., gradient descent), the model parameters of the first model are adjusted to correct the second model. Thus, during the training of the second model, optimization is performed based on the second loss function, making the third feature map output by the second model approximate the label data corresponding to the sample image, thereby improving the target detection accuracy of the second model. After correcting the second model, one training cycle of the second model is completed. At this point, it can be determined whether the function value of the second loss function is less than the first threshold. If so, the trained second model is obtained; otherwise, the next training cycle of the second model continues until the function value of the second loss function is less than the second threshold. The first threshold used to constrain the training of the first model and the second threshold used to constrain the training of the second model can be the same threshold or different thresholds. For example, the second threshold can be greater than the first threshold, thus allowing the training of the second model to be constrained by a smaller threshold on a computing device, improving the target detection accuracy of the second model, and consequently improving the target detection accuracy of the first model.

[0130] In some embodiments, based on the model structure of the second model including a feature extraction layer, a feature decoding layer, a second attention map branch, and a second density map branch, the label data corresponding to the sample image includes the actual attention map and the actual density map of the sample image. The second loss function also includes a second attention map branch loss function and a second density map branch loss function. In this case, a possible implementation of S402 includes: inputting the third feature map into the second attention map branch and the second density map branch respectively to obtain the second attention map and the second density map; determining the function values ​​of the second attention map branch loss function and the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map. Thus, based on the function values ​​of the second attention map branch loss function and the second density map branch loss function, the training process of the second model is guided, improving the accuracy of the second attention map and the second density map output by the second model. This allows the second attention map output by the second model to gradually approach the actual attention map of the sample image, and the second density map output by the second model to gradually approach the actual density map of the sample image, through continuous training.

[0131] Optionally, in determining the function values ​​of the second attention map branch loss function and the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map, the function values ​​of the second attention map branch loss function and the second density map branch loss function can be determined based on the differences between the second attention map and the actual attention map. For example, the mean of the pixel differences between the second attention map and the actual attention map can be determined as the function value of the second attention map branch loss function, and the mean of the pixel differences between the second density map and the actual density map can be determined as the function value of the second density map branch loss function.

[0132] Optionally, in the process of determining the function values ​​of the second attention map branch loss function and the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map, the second attention map and the second density map can be fused (the specific fusion method can be referred to the description in the foregoing embodiments) to obtain the second object detection map of the sample image. The function value of the second attention map branch loss function is determined based on the difference between the second attention map and the actual attention map. The function value of the second density map branch loss function is determined based on the difference between the second object detection map and the actual density map. This makes the second attention map of the sample image output by the second model approximate the actual attention map of the sample image, and the second object detection map of the sample image output by the second model approximate the actual density map of the sample image, thereby improving the accuracy of the object detection map output by the second model.

[0133] For example, the image features output from the feature decoding layer in the second model are input into the second attention map branch, and the resulting second attention map can be represented as: AT = f T (X T The image features output from the feature decoding layer in the second model are input into the second density map branch, resulting in the second density map, which can be represented as: DT = g T (X T The second target detection map of the sample image can be obtained by performing a dot product operation on the second interest map and the second density map. The calculation formula for the second target detection map can be expressed as: DT F =σ(AT)⊙DT.

[0134] Among them, X T X represents the image features output by the feature decoding layer in the second model. T DT represents the second attention map of the sample image, and DT represents the second density map of the sample image. F The second object detection map represents the sample image.

[0135] Furthermore, the formula for the first attention map branch loss function can be expressed as:

[0136]

[0137] Where N represents the number of sample images during one training cycle of the second model, AT i This represents the second attention map of the i-th sample image obtained after processing by the second model. L represents the actual attention map of the i-th sample image. att Let represent the second attention map branch loss function. Therefore, by averaging the differences between the second attention map and the actual attention map of all sample images during one training cycle of the second model, the function value of the second attention map branch loss function is obtained, thus improving the accuracy of the difference between the second attention map and the actual attention map of the sample images reflected by the second attention map branch loss function.

[0138] Optionally, the formula for the first density map branch loss function can be expressed as:

[0139] in, This represents the second object detection map of the i-th sample image obtained after processing by the second model. L represents the actual density map of the i-th sample image. den Let represent the second density map branch loss function. Therefore, by averaging the differences between the second object detection map and the actual density map for all samples during one training cycle of the second model, the function value of the second density map branch loss function is obtained, thus improving the accuracy of the second density map branch loss in reflecting the differences between the second object detection map and the actual density map of the sample images.

[0140] Based on the above embodiments, it can be seen that the second loss function may include a second attention map branch loss function and / or a second density map branch loss function.

[0141] When the second loss function includes a second attention map branch loss function and a second density map branch loss function, one possible implementation of S403 includes: weighted summing of the function values ​​of the second attention map branch loss function and the second density map branch loss function to obtain the final loss value of the second model; and correcting the second model based on the final loss value. Thus, by combining the function values ​​of the second attention map branch loss function and the second density map branch loss function obtained based on sample data and the second model after training, and correcting the second model based on the final loss value obtained after combination, the training effect of the second model is improved in multiple ways.

[0142] Optional, the overall optimization objective of the second model is L T It can be represented as:

[0143] Where q1 and q2 are preset weights. Furthermore, Based on this optimization objective, the final loss value of the second model can be obtained.

[0144] Therefore, based on the above embodiments, a second model can be trained, and based on the trained second model, a first model can be trained. The trained first model is then applied to object detection in images. The trained first model is a lightweight model that can be deployed on lightweight devices. Thus, on the one hand, the efficiency of object detection by the first model is improved, enabling real-time object detection on lightweight devices; on the other hand, the accuracy of object detection by the first model is improved through a structured loss function that reflects the difference between the output of the output layer of the first model and the output of the output layer of the second model, and / or a channel distillation loss function that reflects the difference between the output of the intermediate layer of the first model and the intermediate layer of the second model.

[0145] Reference Figure 5 , Figure 5 Example diagrams illustrating the model structures of the first and second models provided in embodiments of this disclosure. Figure 5 As shown, the first model and the second model are deep learning models, each containing multiple network layers. It can be seen that the model structure of the first model is the same as that of the second model, but the width of the network layers in the first model is smaller than that in the second model. In other words, the number of channels in the network layers of the first model is smaller than that in the second model. Therefore, the second model can extract more image features compared to the first model.

[0146] like Figure 5 As shown, in the first model, features are first extracted from the input image through a network layer for feature extraction (i.e., the feature extraction layer mentioned in the above embodiments), wherein, Figure 5 The diagram shows four feature extraction layers. Following the feature extraction layers is the feature decoding layer, which inputs the extracted image features. Multiple feature decoding layers perform 1x1 convolutions on the image features to obtain the feature map of the input image. Next, the feature map is input into the attention map branch and the density map branch, respectively, to obtain the attention map and the density map. The attention map and the density map are then fused to obtain the attention-bearing density map, which is the target detection map of the input image. The image processing procedure of the second model can be referred to the image processing procedure of the first model, and will not be repeated here.

[0147] like Figure 5As shown, there is a 1*1 convolution between the first model and the second model. This 1*1 convolution is the channel convolution layer in the above embodiment. The channel convolution layer establishes a mapping relationship between the image features output by the feature decoding layer of the second model and the image features output by the feature decoding layer of the second model. Based on this mapping relationship, the channel distillation loss, that is, the function value of the channel distillation loss function, is calculated. The specific calculation process can be referred to the above embodiment.

[0148] like Figure 5 As shown, based on the density map of interest output by the first model and the density map of interest output by the second model, the structured similarity loss, that is, the function value of the structured similarity loss function, can be calculated. The specific calculation process can be referred to the aforementioned embodiment.

[0149] Corresponding to the target detection method in the above embodiments, Figure 6 This is a structural block diagram of a target detection device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 6 The target detection device includes: an acquisition unit 601, a detection unit 602, and a generation unit 603.

[0150] Acquisition unit 601 is used to acquire the image to be detected;

[0151] Detection unit 602 is used to detect targets in an image using a first model to obtain an interest map and a density map of the image;

[0152] The generation unit 603 is used to generate a target detection map of the image based on the attention map and the density map.

[0153] The first model is a lightweight deep learning model trained based on sample data, the second model, and the first loss function. The first loss function includes a distillation loss function that reflects the distillation loss between the first model and the second model. The model size of the second model is larger than that of the first model.

[0154] In one embodiment of this disclosure, the sample data includes sample images and corresponding label data. A training process of the first model includes: extracting and decoding features from the sample images using the first model; extracting and decoding features from the sample images using the second model; determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model; and correcting the first model based on the loss value of the first loss function.

[0155] In one embodiment of this disclosure, the distillation loss function includes a structural similarity loss function. The function value of the first loss function is determined based on the image features extracted by the first model and the image features extracted by the second model. This includes determining the function value of the structural similarity loss function based on the difference between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model.

[0156] In one embodiment of this disclosure, determining the function value of the structural similarity loss function based on the difference between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model includes: determining the structural similarity between the first feature map and the second feature map, wherein the structural similarity includes at least one of the following: brightness similarity, contrast similarity, and pixel vector similarity; and determining the function value of the structural similarity loss function based on the structural similarity.

[0157] In one embodiment of this disclosure, determining the structural similarity between a first feature map and a second feature map includes: dividing the first feature map and the second feature map into multiple image regions respectively; determining the structural similarity between the multiple image regions of the first feature map and the multiple image regions of the second feature map; and determining the structural similarity between the first feature map and the second feature map based on the structural similarity between the multiple image regions of the first feature map and the multiple image regions of the second feature map.

[0158] In one embodiment of this disclosure, the distillation loss function further includes a channel distillation loss function. A channel convolutional layer is connected between the intermediate layer of the first model and the intermediate layer of the second model. The channel convolutional layer is used to establish a mapping relationship between the image features output by the intermediate layer of the first model and the image features output by the intermediate layer of the second model. Determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model further includes: determining the function value of the channel distillation loss function based on the difference between the mutually mapped image features in the image features output by the intermediate layer of the first model and the image features output by the intermediate layer of the second model.

[0159] In one embodiment of this disclosure, the function value of the channel distillation loss function is determined based on the differences between the mutually mapped image features in the image features output from the intermediate layer of the first model and the image features output from the intermediate layer of the second model. This includes: calculating the difference between each pair of mutually mapped image features using the L2 paradigm; and determining the function value of the channel distillation loss function as the sum of the differences between all mutually mapped image features.

[0160] In one embodiment of this disclosure, the label data corresponding to the sample image includes the actual attention map and the actual density map of the sample image. The first model includes a first attention map branch and a first density map branch. The first loss function further includes a feature extraction layer, a feature decoding layer, a first attention map branch loss function, and a first density map branch loss function. Determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model includes: inputting the image features output by the feature decoding layer in the first model into the first attention map branch and the first density map branch respectively to obtain the first attention map and the first density map; and determining the function value of the first attention map branch loss function and the function value of the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map.

[0161] In one embodiment of this disclosure, determining the function values ​​of the first attention map branch loss function and the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map includes: fusing the first attention map and the first density map to obtain a first target detection map of the sample image; determining the function value of the first attention map branch loss function based on the difference between the first attention map and the actual attention map; and determining the function value of the first density map branch loss function based on the difference between the first target detection map and the actual density map.

[0162] In one embodiment of this disclosure, the first attention map branch loss function is:

[0163]

[0164] Where N represents the number of sample images used in one training cycle of the first model, and A i This represents the first attention map of the i-th sample image obtained after processing by the first model. Let represent the actual attention map of the i-th sample image, and σ() represent the activation function;

[0165] And / or, the first density map branch loss function is:

[0166]

[0167] in, This represents the first target detection map of the i-th sample image after processing by the first model. This represents the actual density map of the i-th sample image.

[0168] In one embodiment of this disclosure, the first model is modified based on the loss value of the first loss function, including: weighting and summing the function values ​​of the first interest map branch loss function, the first density map branch loss function, and the distillation loss function to obtain the final loss value of the first model; and modifying the first model based on the final loss value of the first model.

[0169] In one embodiment of this disclosure, the sample image includes the sample image and the label data corresponding to the sample image. A training process of the second model includes: extracting and decoding features from the sample image using the second model; determining the function value of the second loss function based on the third feature map output by the second model and the label data corresponding to the sample image; and correcting the second model based on the function value of the second loss function.

[0170] In one embodiment of this disclosure, the label data corresponding to the sample image includes the actual attention map and the actual density map of the sample image. The second model includes a second attention map branch and a second density map branch. The second loss function includes a second attention map branch loss function and a second density map branch loss function. Determining the function value of the second loss function based on the third feature map output from the second model and the label data corresponding to the sample image includes: inputting the third feature map into the second attention map branch and the second density map branch respectively to obtain the second attention map and the second density map; and determining the function value of the second attention map branch loss function and the function value of the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map.

[0171] In one embodiment of this disclosure, determining the function values ​​of the second attention map branch loss function and the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map includes: fusing the second attention map and the second density map to obtain a second target detection map of the sample image; determining the function value of the second attention map branch loss function based on the difference between the second attention map and the actual attention map; and determining the function value of the second density map branch loss function based on the difference between the second target detection map and the actual density map.

[0172] In one embodiment of this disclosure, the second model is modified based on the function value of the second loss function, including: weighting and summing the function values ​​of the second attention map branch loss function and the second density map branch loss function to obtain the final loss value of the second model; and modifying the second model based on the final loss value of the second model.

[0173] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0174] refer to Figure 7 The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of the present disclosure. The electronic device 700 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0175] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0176] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0177] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0178] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0179] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0180] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0181] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0182] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0183] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0184] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0185] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0186] In a first aspect, according to one or more embodiments of this disclosure, a target detection method is provided, comprising: acquiring an image to be detected; detecting targets in the image using a first model to obtain an attention map and a density map of the image; generating a target detection map of the image based on the attention map and the density map, wherein the target detection map includes a location identifier of the target in the image; wherein the first model is a lightweight deep learning model trained based on sample data, a second model, and a first loss function, the first loss function including a distillation loss function for reflecting the distillation loss between the first model and the second model, and the model size of the second model is larger than the model size of the first model.

[0187] According to one or more embodiments of this disclosure, the sample data includes sample images and label data corresponding to the sample images. A training process of the first model includes: performing feature extraction and decoding on the sample images using the first model; performing feature extraction and decoding on the sample images using the second model; determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model; and correcting the first model based on the loss value of the first loss function.

[0188] According to one or more embodiments of this disclosure, the distillation loss function includes a structural similarity loss function, and determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model includes: determining the function value of the structural similarity loss function based on the difference between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model.

[0189] According to one or more embodiments of this disclosure, determining the function value of the structural similarity loss function based on the difference between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model includes: determining the structural similarity between the first feature map and the second feature map, wherein the structural similarity includes at least one of the following: brightness similarity, contrast similarity, and pixel vector similarity; and determining the function value of the structural similarity loss function based on the structural similarity.

[0190] According to one or more embodiments of this disclosure, determining the structural similarity between the first feature map and the second feature map includes: dividing the first feature map and the second feature map into multiple image regions respectively; determining the structural similarity between the multiple image regions of the first feature map and the multiple image regions of the second feature map; and determining the structural similarity between the first feature map and the second feature map based on the structural similarity between the multiple image regions of the first feature map and the multiple image regions of the second feature map.

[0191] According to one or more embodiments of this disclosure, the distillation loss function further includes a channel distillation loss function, wherein a channel convolutional layer is connected between the intermediate layer of the first model and the intermediate layer of the second model, and the channel convolutional layer is used to establish a mapping relationship between the image features output by the intermediate layer of the first model and the image features output by the intermediate layer of the second model; the step of determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model further includes: determining the function value of the channel distillation loss function based on the difference between the mutually mapped image features in the image features output by the intermediate layer of the first model and the image features output by the intermediate layer of the second model.

[0192] According to one or more embodiments of this disclosure, determining the function value of the channel distillation loss function based on the difference between mutually mapped image features in the image features output from the intermediate layer of the first model and the image features output from the intermediate layer of the second model includes: calculating the difference between each pair of mutually mapped image features using the L2 paradigm; and determining the function value of the channel distillation loss function as the sum of the differences between all mutually mapped image features.

[0193] According to one or more embodiments of this disclosure, the label data corresponding to the sample image includes the actual attention map and the actual density map of the sample image. The first model includes a first attention map branch and a first density map branch, and the first loss function further includes a first attention map branch loss function and a first density map branch loss function. Determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model includes: inputting the first feature map output by the output layer of the first model into the first attention map branch and the first density map branch respectively to obtain the first attention map and the first density map; and determining the function value of the first attention map branch loss function and the function value of the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map.

[0194] According to one or more embodiments of this disclosure, determining the function value of the first attention map branch loss function and the function value of the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map respectively includes: fusing the first attention map and the first density map to obtain a first target detection map of the sample image; determining the function value of the first attention map branch loss function based on the difference between the first attention map and the actual attention map; and determining the function value of the first density map branch loss function based on the difference between the first target detection map and the actual density map.

[0195] In one embodiment of this disclosure, the first attention graph branch loss function is:

[0196]

[0197] Wherein, N represents the number of sample images used in one training process of the first model, and A i This represents the first attention map of the i-th sample image obtained after processing by the first model. Let represent the actual attention map of the i-th sample image, and σ() represent the activation function;

[0198] And / or, the first density map branch loss function is:

[0199]

[0200] Among them, the This represents the first target detection map of the i-th sample image after processing by the first model. This represents the actual density map of the i-th sample image.

[0201] According to one or more embodiments of this disclosure, the step of correcting the first model based on the loss value of the first loss function includes: performing a weighted summation of the function values ​​of the first interest map branch loss function, the first density map branch loss function, and the distillation loss function to obtain the final loss value of the first model; and correcting the first model based on the final loss value of the first model.

[0202] According to one or more embodiments of this disclosure, the sample image includes a sample image and label data corresponding to the sample image. A training process of the second model includes: extracting and decoding features from the sample image using the second model; determining the function value of a second loss function based on a third feature map output by the second model and the label data corresponding to the sample image; and correcting the second model based on the function value of the second loss function.

[0203] According to one or more embodiments of this disclosure, the label data corresponding to the sample image includes the actual attention map and the actual density map of the sample image; the second model includes a second attention map branch and a second density map branch; and the second loss function includes a second attention map branch loss function and a second density map branch loss function. Determining the function value of the second loss function based on the third feature map output from the second model and the label data corresponding to the sample image includes: inputting the third feature map into the second attention map branch and the second density map branch respectively to obtain a second attention map and a second density map; and determining the function values ​​of the second attention map branch loss function and the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map.

[0204] According to one or more embodiments of this disclosure, determining the function values ​​of the second attention map branch loss function and the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map respectively includes: fusing the second attention map and the second density map to obtain a second target detection map of the sample image; determining the function value of the second attention map branch loss function based on the difference between the second attention map and the actual attention map; and determining the function value of the second density map branch loss function based on the difference between the second target detection map and the actual density map.

[0205] According to one or more embodiments of this disclosure, the step of correcting the second model based on the function value of the second loss function includes: performing a weighted summation of the function values ​​of the second attention map branch loss function and the second density map branch loss function to obtain the final loss value of the second model; and correcting the second model based on the final loss value of the second model.

[0206] Secondly, according to one or more embodiments of this disclosure, a target detection device is provided, comprising:

[0207] The acquisition unit is used to acquire the image to be detected;

[0208] A detection unit is used to detect targets in the image using a first model, and to obtain an interest map and a density map of the image.

[0209] A generation unit is configured to generate a target detection map of the image based on the attention map and the density map;

[0210] The first model is a lightweight deep learning model trained based on sample data, a second model, and a first loss function. The first loss function includes a distillation loss function that reflects the distillation loss between the first model and the second model. The model size of the second model is larger than that of the first model.

[0211] According to one or more embodiments of this disclosure, the sample data includes sample images and label data corresponding to the sample images. A training process of the first model includes: performing feature extraction and decoding on the sample images using the first model; performing feature extraction and decoding on the sample images using the second model; determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model; and correcting the first model based on the loss value of the first loss function.

[0212] According to one or more embodiments of this disclosure, the distillation loss function includes a structural similarity loss function, and determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model includes: determining the function value of the structural similarity loss function based on the difference between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model.

[0213] According to one or more embodiments of this disclosure, determining the function value of the structural similarity loss function based on the difference between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model includes: determining the structural similarity between the first feature map and the second feature map, wherein the structural similarity includes at least one of the following: brightness similarity, contrast similarity, and pixel vector similarity; and determining the function value of the structural similarity loss function based on the structural similarity.

[0214] According to one or more embodiments of this disclosure, determining the structural similarity between the first feature map and the second feature map includes: dividing the first feature map and the second feature map into multiple image regions respectively; determining the structural similarity between the multiple image regions of the first feature map and the multiple image regions of the second feature map; and determining the structural similarity between the first feature map and the second feature map based on the structural similarity between the multiple image regions of the first feature map and the multiple image regions of the second feature map.

[0215] According to one or more embodiments of this disclosure, the distillation loss function further includes a channel distillation loss function, wherein a channel convolutional layer is connected between the intermediate layer of the first model and the intermediate layer of the second model, and the channel convolutional layer is used to establish a mapping relationship between the image features output by the intermediate layer of the first model and the image features output by the intermediate layer of the second model; the step of determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model further includes: determining the function value of the channel distillation loss function based on the difference between the mutually mapped image features in the image features output by the intermediate layer of the first model and the image features output by the intermediate layer of the second model.

[0216] According to one or more embodiments of this disclosure, determining the function value of the channel distillation loss function based on the difference between mutually mapped image features in the image features output from the intermediate layer of the first model and the image features output from the intermediate layer of the second model includes: calculating the difference between each pair of mutually mapped image features using the L2 paradigm; and determining the function value of the channel distillation loss function as the sum of the differences between all mutually mapped image features.

[0217] According to one or more embodiments of this disclosure, the label data corresponding to the sample image includes the actual attention map and the actual density map of the sample image. The first model includes a first attention map branch and a first density map branch, and the first loss function further includes a first attention map branch loss function and a first density map branch loss function. Determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model includes: inputting the first feature map output by the output layer of the first model into the first attention map branch and the first density map branch respectively to obtain the first attention map and the first density map; and determining the function value of the first attention map branch loss function and the function value of the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map.

[0218] According to one or more embodiments of this disclosure, determining the function value of the first attention map branch loss function and the function value of the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map respectively includes: fusing the first attention map and the first density map to obtain a first target detection map of the sample image; determining the function value of the first attention map branch loss function based on the difference between the first attention map and the actual attention map; and determining the function value of the first density map branch loss function based on the difference between the first target detection map and the actual density map.

[0219] In one embodiment of this disclosure, the first attention graph branch loss function is:

[0220]

[0221] Wherein, N represents the number of sample images used in one training process of the first model, and A i This represents the first attention map of the i-th sample image obtained after processing by the first model. Let represent the actual attention map of the i-th sample image, and σ() represent the activation function;

[0222] And / or, the first density map branch loss function is:

[0223]

[0224] Among them, the This represents the first target detection map of the i-th sample image after processing by the first model. This represents the actual density map of the i-th sample image.

[0225] According to one or more embodiments of this disclosure, the step of correcting the first model based on the loss value of the first loss function includes: performing a weighted summation of the function values ​​of the first interest map branch loss function, the first density map branch loss function, and the distillation loss function to obtain the final loss value of the first model; and correcting the first model based on the final loss value of the first model.

[0226] According to one or more embodiments of this disclosure, the sample image includes a sample image and label data corresponding to the sample image. A training process of the second model includes: extracting and decoding features from the sample image using the second model; determining the function value of a second loss function based on a third feature map output by the second model and the label data corresponding to the sample image; and correcting the second model based on the function value of the second loss function.

[0227] According to one or more embodiments of this disclosure, the label data corresponding to the sample image includes the actual attention map and the actual density map of the sample image; the second model includes a second attention map branch and a second density map branch; and the second loss function includes a second attention map branch loss function and a second density map branch loss function. Determining the function value of the second loss function based on the third feature map output from the second model and the label data corresponding to the sample image includes: inputting the third feature map into the second attention map branch and the second density map branch respectively to obtain a second attention map and a second density map; and determining the function values ​​of the second attention map branch loss function and the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map.

[0228] According to one or more embodiments of this disclosure, determining the function values ​​of the second attention map branch loss function and the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map respectively includes: fusing the second attention map and the second density map to obtain a second target detection map of the sample image; determining the function value of the second attention map branch loss function based on the difference between the second attention map and the actual attention map; and determining the function value of the second density map branch loss function based on the difference between the second target detection map and the actual density map.

[0229] According to one or more embodiments of this disclosure, the step of correcting the second model based on the function value of the second loss function includes: performing a weighted summation of the function values ​​of the second attention map branch loss function and the second density map branch loss function to obtain the final loss value of the second model; and correcting the second model based on the final loss value of the second model.

[0230] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0231] The memory stores computer-executed instructions;

[0232] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the target detection method as described in the first aspect and various possible designs of the first aspect.

[0233] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the target detection method described in the first aspect and various possible designs of the first aspect is implemented.

[0234] Fifthly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0235] The memory stores computer-executed instructions;

[0236] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the target detection method as described in the first aspect and various possible designs of the first aspect.

[0237] In a sixth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the target detection method described in the first aspect and various possible designs of the first aspect.

[0238] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0239] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0240] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A target detection method, comprising: Acquire the image to be detected; The first model is used to detect targets in the image, and the attention map and density map of the image are obtained. The pixels of the attention map are fused into the pixels of the density map to generate the target detection map of the image, which includes the location identifier of the target in the image; Wherein, the first model is a lightweight deep learning model trained based on sample data, a second model, and a first loss function. The first loss function includes a distillation loss function to reflect the distillation loss between the first model and the second model. The distillation loss function includes a structural similarity loss function and a channel distillation loss function. The function value of the structural similarity loss function is determined based on the structural similarity between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model. The structural similarity includes at least one of the following: brightness similarity, contrast similarity, and pixel vector similarity. The model size of the second model is larger than that of the first model. The channel distillation loss function is used to reflect the difference between the output of the intermediate layer of the first model and the output of the intermediate layer of the second model.

2. The target detection method according to claim 1, wherein the sample data includes sample images and label data corresponding to the sample images, and one training process of the first model includes: The sample image is feature extracted and decoded using the first model; The sample image is then subjected to feature extraction and decoding using the second model. The value of the first loss function is determined based on the image features extracted by the first model and the image features extracted by the second model. The first model is modified based on the loss value of the first loss function.

3. The target detection method according to claim 2, wherein determining the function value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model includes: The function value of the structural similarity loss function is determined based on the difference between the first feature map output by the output layer in the first model and the second feature map output by the output layer in the second model.

4. The target detection method according to claim 3, wherein determining the function value of the structural similarity loss function based on the difference between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model includes: Determine the structural similarity between the first feature map and the second feature map; Based on the structural similarity, the function value of the structural similarity loss function is determined.

5. The target detection method according to claim 4, wherein determining the structural similarity between the first feature map and the second feature map includes: The first feature map and the second feature map are respectively divided into multiple image regions; Determine the structural similarity between multiple image regions of the first feature map and multiple image regions of the second feature map; The structural similarity between the first feature map and the second feature map is determined based on the structural similarity between multiple image regions of the first feature map and multiple image regions of the second feature map.

6. The target detection method according to claim 3, wherein a channel convolutional layer is connected between the intermediate layer of the first model and the intermediate layer of the second model, and the channel convolutional layer is used to establish a mapping relationship between the image features output by the intermediate layer of the first model and the image features output by the intermediate layer of the second model; The step of determining the value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model further includes: Based on the differences between the mutually mapped image features, the function value of the channel distillation loss function is determined from the image features output by the intermediate layer of the first model and the image features output by the intermediate layer of the second model.

7. The target detection method according to claim 6, wherein determining the function value of the channel distillation loss function based on the difference between the mutually mapped image features output from the intermediate layer of the first model and the intermediate layer of the second model includes: The difference between each pair of mutually mapped image features is calculated using the L2 paradigm. The function value of the channel distillation loss function is determined to be the sum of the differences between all mutually mapped image features.

8. The target detection method according to any one of claims 2-7, wherein the label data corresponding to the sample image includes the actual attention map of the sample image and the actual density map of the sample image, the first model includes a feature extraction layer, a feature decoding layer, a first attention map branch and a first density map branch, and the first loss function further includes a first attention map branch loss function and a first density map branch loss function; The step of determining the value of the first loss function based on the image features extracted by the first model and the image features extracted by the second model includes: The image features output by the feature decoding layer in the first model are respectively input into the first attention map branch and the first density map branch to obtain the first attention map and the first density map. Based on the first attention map, the first density map, the actual attention map, and the actual density map, the function values ​​of the branch loss function of the first attention map and the branch loss function of the first density map are determined respectively.

9. The target detection method according to claim 8, wherein determining the function value of the first attention map branch loss function and the function value of the first density map branch loss function based on the first attention map, the first density map, the actual attention map, and the actual density map respectively comprises: The first interest map and the first density map are fused to obtain the first target detection map of the sample image; Based on the difference between the first attention map and the actual attention map, the function value of the branch loss function of the first attention map is determined; The function value of the first density map branch loss function is determined based on the difference between the first target detection map and the actual density map.

10. The target detection method according to claim 9, wherein the first interest map branch loss function is: ； in, N represents the number of sample images used in one training process of the first model. This represents the first attention map of the i-th sample image obtained after processing by the first model. This represents the actual attention map of the i-th sample image, the... Indicates the activation function; And / or, the first density map branch loss function is: ； Among them, the This represents the first target detection map of the i-th sample image after processing by the first model. This represents the actual density map of the i-th sample image.

11. The target detection method according to claim 8, wherein correcting the first model based on the loss value of the first loss function includes: The final loss value of the first model is obtained by weighted summing of the function values ​​of the first attention map branch loss function, the first density map branch loss function, and the distillation loss function. Based on the final loss value of the first model, the first model is modified.

12. The target detection method according to any one of claims 1-7 and 9-11, wherein the sample data includes sample images and label data corresponding to the sample images, and one training process of the second model includes: The sample image is then subjected to feature extraction and decoding using the second model. The value of the second loss function is determined based on the third feature map output from the second model and the label data corresponding to the sample image. The second model is modified based on the value of the second loss function.

13. The target detection method according to claim 12, wherein the label data corresponding to the sample image includes the actual attention map of the sample image and the actual density map of the sample image, the second model includes a second attention map branch and a second density map branch, and the second loss function includes a second attention map branch loss function and a second density map branch loss function; The step of determining the value of the second loss function based on the third feature map output from the second model and the label data corresponding to the sample image includes: The third feature map is input into the second interest map branch and the second density map branch respectively to obtain the second interest map and the second density map; Based on the second attention map, the second density map, the actual attention map, and the actual density map, the function values ​​of the branch loss function of the second attention map and the branch loss function of the second density map are determined respectively.

14. The target detection method according to claim 13, wherein determining the function value of the second attention map branch loss function and the function value of the second density map branch loss function based on the second attention map, the second density map, the actual attention map, and the actual density map respectively comprises: The second interest map and the second density map are fused to obtain the second target detection map of the sample image; Based on the difference between the second attention map and the actual attention map, the function value of the branch loss function of the second attention map is determined; The function value of the second density map branch loss function is determined based on the difference between the second target detection map and the actual density map.

15. The target detection method according to claim 13 or 14, wherein correcting the second model based on the function value of the second loss function includes: The final loss value of the second model is obtained by weighted summing of the function values ​​of the second attention map branch loss function and the second density map branch loss function. Based on the final loss value of the second model, the second model is modified.

16. A target detection device, comprising: The acquisition unit is used to acquire the image to be detected; A detection unit is used to detect targets in the image using a first model, and to obtain an interest map and a density map of the image. A generation unit is used to fuse each pixel of the interest map into each pixel of the density map to generate a target detection map of the image. Wherein, the first model is a lightweight deep learning model trained based on sample data, a second model, and a first loss function. The first loss function includes a distillation loss function to reflect the distillation loss between the first model and the second model. The distillation loss function includes a structural similarity loss function and a channel distillation loss function. The function value of the structural similarity loss function is determined based on the structural similarity between the first feature map output by the output layer of the first model and the second feature map output by the output layer of the second model. The structural similarity includes at least one of the following: brightness similarity, contrast similarity, and pixel vector similarity. The model size of the second model is larger than that of the first model. The channel distillation loss function is used to reflect the difference between the output of the intermediate layer of the first model and the output of the intermediate layer of the second model.

17. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the target detection method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the target detection method as described in any one of claims 1 to 15.

19. A computer program product, characterized in that, The computer program product includes computer execution instructions, which, when executed by a processor, implement the target detection method as described in any one of claims 1 to 15.

Images