Target detection method and equipment

Abstract

The disclosed embodiment provides a target detection method and device, the method comprising: obtaining an image to be detected; in a target detection model, encoding the image through a coding network to obtain target image features of the image; in the target detection model, decoding the target image features through a first network branch and a second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other through a relationship modeling network for image features; generating a target detection map of the image based on the attention map and the density map. Thus, through the relationship modeling network between network branches, the mutual exchange of information between network branches is promoted, guiding each other to learn better image features, and improving the accuracy of target detection.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular, to a target detection method and device. Background Art

[0002] As business develops, it is necessary to realize target detection in complex scenes, such as pedestrian detection in street scenes and vehicle detection in road scenes.

[0003] Currently, target detection can be performed using detectors based on the Histogram of Oriented Gradient (HOG) or detectors based on deep learning. However, this method is prone to false detection and missed detection when the target is severely occluded.

[0004] Therefore, the accuracy of target detection needs to be improved. Summary of the invention

[0005] The embodiments of the present disclosure provide a method and device for object detection to overcome the problem that the accuracy of object detection in an image needs to be improved.

[0006] In a first aspect, an embodiment of the present disclosure provides a target detection method, including:

[0007] Acquire an image to be detected;

[0008] In the target detection model, the image is encoded by an encoding network to obtain target image features of the image;

[0009] In the target detection model, the target image features are decoded by a first network branch and a second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network;

[0010] An object detection map of the image is generated according to the attention map and the density map, wherein the object detection map includes position identifiers of objects in the image.

[0011] In a second aspect, an embodiment of the present disclosure provides a model determination method, including:

[0012] Acquire a training sample, wherein the training sample includes a sample image, an actual attention map corresponding to the sample image, and an actual density map corresponding to the sample image;

[0013] In the target detection model, the sample image is encoded by a coding network to obtain target image features of the sample image;

[0014] In the target detection model, the target image features are decoded by a first network branch and a second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network;

[0015] Generating a target detection map corresponding to the sample image according to the attention map and the density map;

[0016] The target detection model is adjusted according to the first loss function, the second loss function, the sample image, the target detection map, the actual attention map and the actual density map.

[0017] In a third aspect, an embodiment of the present disclosure provides a target detection device, including:

[0018] An acquisition unit, used for acquiring an image to be detected;

[0019] An encoding unit, used to encode the image through an encoding network in a target detection model to obtain target image features of the image;

[0020] A decoding unit, configured to decode the target image features through a first network branch and a second network branch in the target detection model to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network;

[0021] A generating unit is used to generate a target detection map of the image according to the attention map and the density map, wherein the target detection map includes a position identifier of the target in the image.

[0022] In a fourth aspect, an embodiment of the present disclosure provides an image determination device, including:

[0023] An acquisition unit, configured to acquire a training sample, wherein the training sample includes a sample image, an actual attention map corresponding to the sample image, and an actual density map corresponding to the sample image;

[0024] An encoding unit, used to encode the sample image through an encoding network in a target detection model to obtain a target image feature of the sample image;

[0025] A decoding unit, configured to decode the target image features through a first network branch and a second network branch in the target detection model to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network;

[0026] A generating unit, configured to generate a target detection map corresponding to the sample image according to the attention map and the density map;

[0027] An adjustment unit is used to adjust the target detection model according to the first loss function, the second loss function, the sample image, the target detection map, the actual attention map and the actual density map.

[0028] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0029] The memory stores computer-executable instructions;

[0030] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the target detection method described in the first aspect or various possible designs of the first aspect, or executes the model determination method described in the second aspect or various possible designs of the second aspect.

[0031] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the target detection method as described in the first aspect or various possible designs of the first aspect is implemented, or the model determination method as described in the second aspect or various possible designs of the second aspect is implemented.

[0032] In the seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, wherein the computer program product includes computer execution instructions. When a processor executes the computer execution instructions, the target detection method as described in the first aspect or various possible designs of the first aspect is implemented, or the model determination method as described in the second aspect or various possible designs of the second aspect is implemented.

[0033] In the target detection method and device provided by the present embodiment, in the target detection model, an image to be detected is encoded through an encoding network, and then decoded through a first network branch and a second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch. Based on the attention map and the density map, a target detection map corresponding to the image is obtained, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network.

[0034] Therefore, in this embodiment, on the one hand, different image features are focused on during the decoding process by the first network branch and the second network branch, and on the other hand, the information exchange between the two network branches is increased by introducing a relational modeling network between the first network branch and the second network branch. The image features extracted by the first network branch can assist the second network branch in learning better image features, and the image features extracted by the second network branch can also assist the first network branch in learning better image features. Thus, the accuracy of target detection is improved through the improvement of the above network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 An example diagram of an application scenario provided by an embodiment of the present disclosure;

[0037] Figure 2 Schematic diagram of the target detection method provided in the embodiment of the present disclosure Figure 1 ;

[0038] Figure 3 Example of a structure of a target detection model provided in the embodiment of the present disclosure Figure 1 ;

[0039] Figure 4a A schematic diagram of a process for decoding target image features through a first network branch and a second network branch provided in an embodiment of the present application;

[0040] Figure 4b A schematic diagram of a process of performing information aggregation processing on a first image feature and a second image feature in a relational modeling network;

[0041] Figure 5 Example of a structure of a target detection model provided in the embodiment of the present disclosure Figure 2 ;

[0042] Figure 6 A diagram showing an example structure of a Res2Net network provided in an embodiment of the present disclosure;

[0043] Figure 7 A flow chart of a model determination method provided in an embodiment of the present disclosure;

[0044] Figure 8A structural block diagram of a target detection device provided in an embodiment of the present disclosure;

[0045] Fig. 9 A structural block diagram of a model determination device provided in an embodiment of the present disclosure;

[0046] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0048] refer to Figure 1 , Figure 1 An example diagram of an application scenario provided by an embodiment of the present disclosure.

[0049] like Figure 1 As shown, the application scenario is, for example, a crowd detection scenario in a road scene, in which the target to be detected is a pedestrian. Specifically, in a road scene, a video is shot by a roadside camera 101, and the camera 101 or a terminal 102 connected to the camera 101 performs real-time pedestrian detection on the image in the video to obtain the number and distribution of pedestrians in the image.

[0050] In the prior art, generally speaking, the following two methods are usually used for crowd detection:

[0051] Method 1: Pedestrian Detection-Based Method

[0052] In this type of method, a traditional detector based on the Histogram of Oriented Gradient (HOG) or a detector based on a deep learning algorithm is used for pedestrian detection. Among them, deep learning algorithms include the YOLOs algorithm or the Region Convolutional Neural Networks (RCNNs). However, the human body in the crowd is severely occluded, which seriously affects the detection accuracy of this type of method, and it is easy to have false detection and missed detection.

[0053] Method 2: Estimation based on population density map

[0054] In this type of method, head detection is performed on the image to provide information about the number of pedestrians and a density map of their distribution. This method is often better than the method based on pedestrian detection: compared with the above detector-based pedestrian detection method, the density map not only provides information about the number of pedestrians, but also reflects their distribution, allowing the crowd detection model to better fit the original image with varying densities. However, the main challenges faced by this method are the different scales of heads in dense crowds (heads closer to the camera appear larger) and complex scene information, which interfere with head detection based on density map estimation, resulting in low accuracy of crowd detection.

[0055] To solve the above problems, the embodiments of the present disclosure provide a method and device for target detection. In the embodiments of the present disclosure, the target detection model includes a coding network, a first network branch, a second network branch and a relationship modeling network, the coding network is used to encode the image, the first network branch is used to decode the image features output by the coding network to obtain an attention map, the second network branch is used to decode the image features output by the coding network to obtain a density map, and the relationship modeling network is used to establish image feature interaction between the first network branch and the second network branch.

[0056] It can be seen that the embodiments of the present disclosure have the following beneficial effects: on the one hand, by focusing on different aspects of image features in target detection through a separate decoding network (i.e., the first network branch and the second network branch), a more accurate attention map and density map are obtained, that is, the idea of ​​multi-task learning (one task is the learning of the attention map, and the other task is the learning of the density map) is used to improve the accuracy of target detection; on the other hand, through the relational modeling network, different decoding networks can exchange information (the information of the first network branch can be transmitted to the second network branch, and the information of the second network branch can be transmitted to the first network branch), to assist each other in learning better image features, that is, in multi-task learning, information transfer between different tasks is promoted, thereby improving the robustness of the target detection model and the target detection accuracy.

[0057] Among them, in the embodiment of the present disclosure, the attention map focuses on the clarity and accuracy of target detection in the image, and can more clearly reflect the position of a single target in the image; the density map focuses on the comprehensiveness of target detection in the image, and can more comprehensively reflect the distribution of multiple targets in the image.

[0058] Optionally, the target detected by the embodiment of the present disclosure may be a person or a vehicle. In particular, the embodiment of the present disclosure may be applied to crowd detection, and may achieve better results in the head counting task of crowd detection, that is, may effectively improve the accuracy of head counting.

[0059] refer to Figure 2 , Figure 2Schematic diagram of the target detection method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a terminal or a server. Figure 2 As shown, the target detection method includes:

[0060] S201, obtaining an image to be detected.

[0061] The number of images to be detected is one or more.

[0062] In one example, the image to be detected may be an image captured by a camera in real time, or one or more frames of images may be obtained from a video captured by a camera in real time. For example, the terminal performs target detection on the captured video while capturing the video.

[0063] In another example, the image to be detected may be an image input by a user or selected by a user. For example, the user inputs the image to be detected on a display interface of the terminal, or selects the image to be detected from a plurality of images displayed on the terminal.

[0064] In another example, the image to be detected may be an image received from another device. For example, the server receives an image sent by a terminal.

[0065] In another example, the image to be detected may be an image played in real time on the terminal. For example, when the terminal detects that an image or video is being played, it obtains the image or video frame being played. Thus, target detection of the image played in real time on the terminal is achieved.

[0066] S202: In the target detection model, the image is encoded through the encoding network to obtain the target image features of the image.

[0067] Among them, the encoding network includes multiple network layers, and the target image features are the image features output by the encoding network after the image is input into the encoding network, that is, the image features output by the last network layer in the encoding network.

[0068] In this step, after obtaining the image to be detected, the image can be directly input into the target detection model, or the image can be preprocessed (such as image denoising, cropping, enhancement, etc.), and the preprocessed image can be input into the target detection model. After the image is input into the target detection model, the image is encoded by the encoding network in the first model to obtain the target image features output by the encoding network.

[0069] S203. In the target detection model, the target image features are decoded by the first network branch and the second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network.

[0070] Among them, the first network branch and the second network branch belong to the decoding network in the target detection model, the first network branch includes multiple network layers, and the second network branch also includes multiple network layers.

[0071] Among them, the attention map focuses on the clarity and accuracy of target detection in the image, and can more clearly reflect the position of a single target in the image. The density map focuses on the comprehensiveness of target detection in the image, and can more comprehensively reflect the distribution of multiple targets in the image. Therefore, the image features that the target detection model needs to pay attention to in the process of generating the attention map are different from those that the target detection model needs to pay attention to in the process of generating the density map, and the optimization direction of the model is also different. By learning the attention map and density map separately through two network branches, the model effect can be effectively improved, and the image quality of the attention map and density map can be improved.

[0072] In this embodiment, the target image features can be input into the first network branch and the second network branch respectively. The target image features are decoded by multiple network layers in the first network branch to obtain the attention map output by the first network branch. The target image features are decoded by multiple network layers in the second network branch to obtain the density map output by the second network branch.

[0073] In this embodiment, during the encoding process, the image features extracted by the first network branch are input into the relational modeling network, and the image features extracted by the second network branch are also input into the relational modeling network. In the relational modeling network, image feature interaction is performed between the first network branch and the second network branch. The specific interaction process may include: adjusting the network features extracted by the second network branch based on the image features extracted by the first network branch, and feeding back the adjusted network features to the second network branch; adjusting the network features extracted by the first network branch based on the image features extracted by the second network branch, and feeding back the adjusted network features to the first network branch.

[0074] S204: Generate an object detection map of the image according to the attention map and the density map, where the object detection map includes position identifiers of objects in the image.

[0075] The target detection map includes the position identification of the target in the image. For example, when the target is a pedestrian, the highlighted point can be used as the position identification of the target in the image. In this way, the number and distribution of multiple highlighted points in the image reflect the number and distribution of multiple targets in the image.

[0076] In this embodiment, after obtaining the attention map and the density map, the attention map and the density map can be combined to obtain a density map with attention, that is, a target detection map. Therefore, while reflecting the density distribution of multiple targets in the image through the target detection map, the accuracy of the target location reflected in the target detection map is improved, thereby improving the clarity, accuracy and comprehensiveness of target detection in the image, and effectively improving the target detection effect.

[0077] The combination of the attention map and the density map is the combination of each pixel in the attention map and each pixel in the density map, that is, each pixel of the attention map is fused into each pixel of the density map.

[0078] In the disclosed embodiment, during the target detection process, the first network branch is responsible for generating an attention map, and the second network branch is responsible for generating a density map, and information exchange between the first network branch and the second network branch is achieved through a relationship modeling module. Thus, by utilizing multi-task learning and promoting information exchange between different tasks, the accuracy of the attention map and the density map generated during the target detection process is improved, thereby improving the accuracy of the target detection map generated based on the attention map and the density map, that is, improving the accuracy of target detection.

[0079] As an example, refer to Figure 3 , Figure 3 Example of a structure of a target detection model provided in the embodiment of the present disclosure Figure 1 .like Figure 3 As shown, in the target detection process, the image to be detected is input into the encoding network, and then decoded by the first network branch to obtain an attention map, and then decoded by the second network branch to obtain a density map, and the target detection map is obtained by combining the attention map and the density map. In the decoding process, the relationship modeling network between the first network branch and the second network branch acts as a bridge, and exchanges information with the first network branch and the second network branch, so that the image features output by the network layer in the first network branch and the image features output by the network layer in the second network branch can enhance each other, and assist the first network branch and the second network branch to learn better image features.

[0080] In some embodiments, there are multiple relationship modeling modules. A relationship modeling network is connected between the one-to-one corresponding network layers in the first network branch and the second network branch, and the relationship modeling network is used to establish the interaction between the image features extracted by the network layer in the first network branch and the image features extracted by the corresponding network layer in the second network branch. There are multiple one-to-one corresponding network layers in the first network branch and the second network branch, and different one-to-one corresponding network layers are connected to different relationship modeling networks.

[0081] Thus, with the help of the one-to-one correspondence between the network layers of the first network branch and the network layers of the second network branch and multiple relationship modeling modules, the interaction of image features between the one-to-one corresponding network layers in the first network branch and the second network branch is realized, the information interaction effect between the first network branch and the second network branch is improved, and then the accuracy of target detection is improved.

[0082] In one example, a relational modeling network is connected between the one-to-one corresponding network layers in the first network branch and the second network branch, including: a relational modeling network is connected between the nth network layer in the first network branch and the nth network layer in the second network branch. The nth network layer in the first network branch and the nth network branch in the second network branch are one-to-one corresponding network layers.

[0083] Specifically, the first network layer in the first network branch is connected to the first network layer in the second network branch by a relational modeling network a1, the second network layer in the first network branch is connected to the second network layer in the second network branch by a relational modeling network a2, and so on.

[0084] In another example, a relational modeling network is connected between the one-to-one corresponding network layers in the first network branch and the second network branch, including: a relational modeling network is connected between the Ni-th network layer in the first network branch and the Mi-th network layer in the second network branch. Among them, the Ni-th network layer in the first network branch and the Mi-th network layer in the second network branch are one-to-one corresponding network layers, N is the total number of network layers in the first network branch, M is the total number of network layers in the second network branch, and i is greater than or equal to 0.

[0085] Specifically, a relational modeling network a1 is connected between the penultimate network layer in the first network branch and the penultimate network layer in the second network branch, a relational modeling network a2 is connected between the penultimate network layer in the first network branch and the penultimate network layer in the second network branch, and so on.

[0086] Based on the example of modeling a network in which the nth network layer in the first network branch and the nth network branch in the second network branch are network layers corresponding to each other in one-to-one manner and the nth network layer in the first network branch and the nth network layer in the second network branch are connected with each other, refer to Figure 4a , Figure 4a A schematic diagram of a process for decoding target image features through a first network branch and a second network branch provided in an embodiment of the present application. Figure 4a As shown, the decoding process includes:

[0087] S401. Decode the input image feature through the nth network layer in the first network branch to obtain a first image feature.

[0088] In this embodiment, in the actual process of decoding the target image features, the target image features are decoded in sequence through each network layer in the first network branch, and the target image features are decoded in sequence through each network layer in the second network branch. For the sake of simplicity, the decoding process of the nth network layer in the first network branch and the decoding process of the nth network layer in the second network branch are described here.

[0089] In this embodiment, in the nth network layer in the first network branch, decoding processing is performed by upsampling the image features input to the nth network layer to obtain the first image features.

[0090] S402. Decode the input image feature through the nth network layer in the second network branch to obtain a second image feature.

[0091] In this embodiment, in the nth network layer in the second network branch, decoding processing is performed by upsampling the image features input to the nth network layer to obtain the second image features.

[0092] In S401 and S402:

[0093] When n is 1, the image features input into the nth network layer in the first network branch and the nth network layer in the second network branch are the target image features from the encoding network.

[0094] When n is greater than 1, the image features of the nth network layer input into the first network branch are obtained by processing the image features extracted from the n-1th network layer in the first network branch through a relationship modeling network between the n-1th network layer in the first network branch and the n-1th network layer in the second network branch; the image features of the nth network layer input into the second network branch are obtained by processing the image features extracted from the n-1th network layer in the second network branch through a relationship modeling network between the n-1th network layer in the first network branch and the n-1th network layer in the second network branch.

[0095] S403, performing information aggregation processing on the first image feature and the second image feature through a relationship modeling network connected to the nth network layer in the first network branch and the nth network layer in the second network branch to obtain a processed first image feature and a processed second image feature.

[0096] In this embodiment, the first image feature and the second image feature are respectively input into the relational modeling network connected to the nth network layer in the first network branch and the nth network layer in the second network branch. In the relational modeling network, the second image feature is adjusted according to the first image feature, and the first image feature is adjusted according to the second image feature, so as to realize information fusion processing of the first image feature and the second image feature, and obtain the processed first image feature and the processed second image feature.

[0097] Among them, the processed first image feature is the image feature of the n+1th network layer input in the first network branch, and the processed second image feature is the image feature of the n+1th network layer input in the second network branch, so as to continue the decoding processing of the image feature in the n+1th network layer of the first network branch and continue the decoding processing of the image feature in the n+1th network layer of the second network branch.

[0098] In some embodiments, reference Figure 4b , Figure 4b FIG. 4 is a flow chart of performing information aggregation processing (i.e., S403) on the first image feature and the second image feature in the relational modeling network. Figure 4b As shown, performing information aggregation processing on the first image feature and the second image feature in the relational modeling network includes (that is, a possible implementation of S403 includes):

[0099] S4031. In the relational modeling network, perform a dot product operation on the first image feature and the second image feature to obtain an aggregated feature of the first image feature and the second image feature.

[0100] S4032: Based on the aggregated features, information enhancement is performed on the first image features and the second image features to obtain enhanced first image features and enhanced second image features.

[0101] S4033. Perform a dot product operation on the first image feature and the enhanced first image feature to obtain a processed first image feature, and perform a dot product operation on the second image feature and the enhanced second image feature to obtain a processed second image feature.

[0102] In S4031, assuming that the first image feature is F a , the second image feature is F b , then the aggregated features of the first image and the second image obtained by the dot product operation can be expressed as .

[0103] In S4032, the first image feature and the second image feature may be adjusted according to the aggregate feature. For example, the aggregate feature may be added to the first image feature and the second image feature. Finally, the enhanced first image feature and the enhanced second image feature are obtained.

[0104] Optionally, a possible implementation method of S4032 includes: in different network branches of the relational modeling network, based on the aggregated features, learning the similarity information between the first image feature and the second image feature, and obtaining the similarity weight factor corresponding to the first image feature and the similarity weight factor corresponding to the second image feature; performing preliminary information enhancement on the first image feature according to the similarity weight factor corresponding to the first image feature, and obtaining the preliminary enhanced first image feature; performing preliminary information enhancement on the second image feature according to the similarity weight factor corresponding to the second image feature, and obtaining the preliminary enhanced second image feature; performing information enhancement on the preliminary enhanced first image feature again through a feature encoding method, and obtaining the enhanced first image feature; performing information enhancement on the preliminary enhanced second image feature again through a feature encoding method, and obtaining the enhanced second image feature.

[0105] Specifically, in this possible implementation, the aggregated features can be divided into two paths in the relational modeling network, one path learning the similarity information between the aggregated features and the first image features to obtain the similarity weight factor corresponding to the first image features, and the other path learning the similarity information between the aggregated features and the second image features to obtain the similarity weight factor corresponding to the second image features. After obtaining the similarity weight factor corresponding to the first image features and the similarity weight factor corresponding to the second image features, the first image features can be adjusted based on the similarity weight factor corresponding to the first image features to achieve preliminary information enhancement of the first image features; the second image features can be adjusted based on the similarity weight factor corresponding to the second image features to achieve preliminary information enhancement of the second image features.

[0106] Optionally, in the process of learning the similarity information between the aggregated features and the first image features, the aggregated features may be convolved through a convolution layer, and then the aggregated features after the convolution operation may be normalized through a normalized network layer (Softmax layer) to obtain a similarity weight factor corresponding to the first image features. Similarly, in the process of learning the similarity information between the aggregated features and the second image features, optionally, the aggregated features may be convolved through a convolution layer, and then the aggregated features after the convolution operation may be normalized through a normalized network (Softmax layer) to obtain a similarity weight factor corresponding to the second image features. The model parameters of the network layer (convolution layer and / or normalization network layer) involved in generating the similarity weight factor corresponding to the first image features are different from those of the network layer (convolution layer and / or normalization network layer) involved in generating the similarity weight factor corresponding to the second image features.

[0107] Furthermore, the similarity weight factor corresponding to the first image feature is expressed as: , the similarity weight factor corresponding to the second image feature is expressed as: Among them, g a1 and g b1 Represents different convolutional layers, for example, different scales of 1 1 convolutional layer.

[0108] Optionally, in the process of performing preliminary information enhancement on the first image feature, a similarity weight factor corresponding to the first image feature may be cross-producted with the first image feature to obtain the first image feature after preliminary enhancement. In the process of performing preliminary information enhancement on the second image feature, a similarity weight factor corresponding to the second image feature may be cross-producted with the second image feature to obtain the second image feature after preliminary enhancement. Thus, by combining the similarity weight factor and the cross-product operation, information enhancement of image features is achieved, and the feature interaction effect between the first image feature and the second image feature is improved.

[0109] Assume that the first image feature after preliminary enhancement is F a2 , the second image feature after preliminary enhancement is F b2 , then the preliminary enhancement of the first image feature and the preliminary enhancement of the second image feature can be expressed as:

[0110] , .

[0111] After obtaining the first image feature after preliminary enhancement and the second image feature after preliminary enhancement, the information of the first image feature after preliminary enhancement is further enhanced through the feature encoding method to obtain the enhanced first image feature; the information of the second image feature after preliminary enhancement is further enhanced through the feature encoding method to obtain the enhanced second image feature. Thus, the information expression of the first image feature and the information expression of the second image feature are further enhanced through the encoding method.

[0112] Optionally, in the relational modeling network, the first image feature and the second image feature after preliminary enhancement are encoded through a convolution layer and an activation function to achieve further information enhancement of the first image feature and the second image feature. The activation function may be a Rectified Linear Unit (ReLU) function.

[0113] Assume that the first image feature after enhancement (i.e., re-enhancement) is F a3 , the second image feature after enhancement (i.e., re-enhancement) is F b3 , through the feature encoding method, the information enhancement of the first image feature after preliminary enhancement and the second image feature after preliminary enhancement can be expressed as:

[0114] , .

[0115] Among them, g a2 , g a3 , g b2 , g b3 Represents different convolutional layers. and ReLU both represent activation functions.

[0116] In S4033, the first image feature and the enhanced first image feature may be subjected to a dot product operation to obtain the processed first image feature, and the second image feature and the enhanced second image feature may be subjected to a dot product operation to obtain the processed second image feature. Assume that the processed first image feature is F af , the processed second image feature is F bf , then the first image feature and the second image feature after processing obtained by dot product operation can be expressed as:

[0117] , .

[0118] Later, F af Input to the n+1th network layer in the first network branch for decoding, and F bfThe data is input to the n+1th network layer in the second network branch for decoding.

[0119] Therefore, through the above-mentioned dot product operation, information enhancement (preliminary information enhancement, secondary information enhancement), and cross product operation, the information exchange effect between the first image feature and the second image feature is effectively improved, that is, the image feature interaction effect between the first network branch and the second network branch is effectively improved, thereby improving the robustness of the target detection model and the target detection accuracy.

[0120] For example, refer to Figure 5 , Figure 5 Example of a structure of a target detection model provided in the embodiment of the present disclosure Figure 2 .like Figure 5 As shown, the target detection process is as follows: 1) After the image is input into the encoder, the target image features are extracted through multiple network layers of the encoder; 2) The target image features are respectively input into the first network branch and the second network branch for decoding processing. During the decoding process, the image features extracted by the first network in the first network branch and the image features extracted by the first network layer in the second network branch are respectively input into the relational modeling network, and then the image features from the first network layer in the first network branch processed in the relational modeling network are input into the second network layer in the first network branch, and the image features from the first network layer in the second network branch processed in the relational modeling network are input into the second network layer in the second network branch. In this way, the interaction of image features of multiple network layers in the first network branch and multiple network layers in the second network branch is realized. Finally, the attention map output by the first network branch and the density map output by the second network branch are obtained, and the target detection map is obtained by combining the attention map and the density map. Among them, Figure 5 The "relational modeling" in refers to the relational modeling network.

[0121] In some embodiments, based on any of the foregoing embodiments, multiple network layers in the encoding network correspond one-to-one to multiple network layers in the first network branch and multiple network layers in the second network branch, respectively, and the image features output by the network layers in the encoding network are used to input the next network layer, the corresponding network layer in the first network branch, and the corresponding network layer in the second network branch. Thus, by using the image features extracted by the multiple network layers in the encoding network as part of the input data of the network layers in the first network branch and the network layers in the second network branch, the feature richness in the decoding process is improved, thereby improving the accuracy of the attention graph learning of the first network branch and the density graph learning of the second network branch.

[0122] For example, Figure 5As shown, the image features extracted by the first network layer in the coding network are used to input the second network layer in the coding network, the third network layer in the first network branch, and the third network layer in the second network branch; the image features extracted by the second network layer in the coding network are used to input the third network layer in the coding network, the second network layer in the first network branch, and the second network layer in the second network branch; the image features extracted by the third network layer in the coding network are used to input the fourth network layer in the coding network, the first network layer in the first network branch, and the first network layer in the second network branch.

[0123] In some embodiments, based on any of the foregoing embodiments, the network layer in the encoding network adopts a residual module, and the residual module includes a Res2net network. Among them, the Res2net network includes a residual block. In the Res2net network, a single residual block is constructed with a hierarchical residual connection. This structure increases the receptive field within the block instead of capturing finer-grained scales of different levels in the image layer by layer, thereby realizing the extraction of multi-scale features and improving the ability of the convolutional neural network to detect targets in the image. Therefore, in this embodiment, by adopting the Res2net network in the encoding network, the extraction of multi-scale features in the encoding process is realized, which is conducive to improving the target detection accuracy of the target detection model.

[0124] In the Res2Net network, the convolutional layer is replaced by a layered convolutional group, and different convolutional groups are connected layer by layer in a similar way to residual connections. In this multi-scale network structure, each time the image feature passes through a convolution kernel, it can obtain a larger receptive field. Moreover, due to the combinatorial explosion effect, the output of the Res2Net network covers a different number of receptive fields of different sizes and scales, as well as different combinations of these receptive fields, which is conducive to extracting local and global information in the image. In addition, in order to better fuse information of different scales, the Res2Net network connects the features of different parts of the residual block in parallel, and then fuses the information through the convolutional layer. This grouping and merging strategy enables the convolutional layer to process image features more effectively and improve the accuracy of target detection in the image.

[0125] Optionally, you can train a Res2Net network on a pre-collected training dataset, and then load the Res2Net network into the encoder of the target detection model and apply it to target detection in images. For example, you can train a Res2Net network on an image classification task or other image processing task, and then continue to train the Res2Net network on an image target detection task.

[0126] As an example, refer to Figure 6 , Figure 6 This is a diagram showing an example structure of a Res2Net network provided in an embodiment of the present disclosure. Figure 6 As shown, in the Res2Net network, first pass 1 1's convolution layer extracts features from the input image to obtain image features, divides the image features, and obtains multiple block features: X1, X2, X3, and X4. Afterwards, these features are processed at different scales: feature X1 is not processed, and feature Y1 is directly obtained; feature X2 is convolved through the first convolution group to obtain feature K2. On the one hand, feature K2 is determined as feature Y2 obtained after processing X2, and on the other hand, feature K2 and feature X3 are input into the second convolution group together to obtain feature K3; similarly, on the one hand, feature K3 is determined as feature Y3 obtained after processing X3, and on the other hand, feature K3 and feature X4 are input into the third convolution group together to obtain feature K4, and feature K4 is determined as feature Y4 obtained after processing X4. Thus, multi-scale feature extraction is achieved, and features Y1, Y2, Y3, and Y4 of different scales are input into 1 1 convolution layer to perform information fusion. Figure 6 3 of 3 3 represents the size of the convolution kernel in the convolution group.

[0127] In some embodiments, based on any of the foregoing embodiments, a target detection map of an image is generated according to an attention map and a density map, including: performing a dot product operation on the attention map and the density map to obtain a density map with attention. Thus, by means of a dot product operation, the combination of each pixel in the attention map and each pixel in the density map is achieved, thereby improving the combination effect of the attention map and the density map. Among them, performing a dot product operation on the attention map and the density map means treating the attention map as a first matrix composed of multiple pixel values, treating the density map as a second matrix composed of multiple pixel values, performing a click operation on the first matrix and the second matrix, and the result of the operation is the density map with attention.

[0128] Furthermore, the attention map is processed by an activation function, and the processed attention map and the density map are subjected to a dot product operation to obtain a density map with attention, thereby improving the fusion effect of the attention map in the density map and improving the accuracy of the number information and density distribution of the targets in the image reflected by the target detection map. The activation function is, for example, a sigmoid function.

[0129] The following describes the training process of the target detection model in any of the aforementioned embodiments through examples.

[0130] It should be noted that the application process of the target detection model for target detection in an image and the training process of the target detection model can be performed on the same device or on different devices.

[0131] refer to Figure 7 , Figure 7The following is a flow chart of a model determination method provided in an embodiment of the present disclosure. Figure 7 As shown, the model determination method includes:

[0132] S701. Obtain training samples, where the training samples include a sample image, an actual attention map corresponding to the sample image, and an actual density map corresponding to the sample image.

[0133] Among them, the actual attention map corresponding to the sample image and the actual density map corresponding to the sample image are used as label data in the training process, which can be obtained by manual annotation or by performing target detection on the sample image through a complex model with high accuracy.

[0134] The training samples can come from user input, can be obtained from a database, or can come from a publicly available image dataset on the Internet.

[0135] S702: In the target detection model, the sample image is encoded through the encoding network to obtain the target image features of the sample image.

[0136] S703. In the target detection model, the target image features are decoded by the first network branch and the second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network.

[0137] S704: Generate an object detection map corresponding to the sample image according to the attention map and the density map.

[0138] The processing process of the sample image in S702-S704 can refer to the process of processing the image to be detected by the target detection model in the aforementioned embodiment, which will not be repeated here.

[0139] S705. Adjust the target detection model according to the first loss function, the second loss function, the sample image, the target detection map, the actual attention map, and the actual density map.

[0140] Among them, the first loss function is the loss function corresponding to the first network branch, and the second loss function is the loss function corresponding to the second network branch.

[0141] In this embodiment, the first loss function is used to determine the difference between the attention map output by the first network branch and the actual attention map corresponding to the sample image, and a first loss value is obtained. The second loss function is used to determine the difference between the density map output by the second network branch and the actual density map corresponding to the sample image, and a second loss value is obtained. Based on the first loss value and the second loss value, an overall loss value is determined. Based on the overall loss value, the target detection model is adjusted. This process is a one-time adjustment process of the target detection model. In order to improve the model effect of the target detection model, the target detection model is usually adjusted multiple times until the overall loss value is less than the loss threshold, or until the number of adjustments is greater than the number threshold. Since each adjustment training is the same, only one training process of the target detection model is described here.

[0142] Among them, adjusting the target detection model includes adjusting the parameters of the encoding network, the first network branch, the second network branch and the relationship modeling network in the target detection model.

[0143] Optionally, multiple sample images may be sampled during a training process. The first loss value corresponding to each sample image and the second loss value corresponding to each sample image are determined in the above manner. The first loss values ​​corresponding to each sample image are averaged to obtain a final first loss value, and the second loss values ​​corresponding to each sample image are averaged to obtain a final second loss value. Afterwards, the overall loss value is determined based on the final first loss value and the final second loss value. Thus, the target detection model can be adjusted based on multiple sample images during a training process to improve the training efficiency of the target detection model.

[0144] Optionally, the first loss function may adopt a cross entropy loss function.

[0145] Furthermore, the first loss function can be expressed as:

[0146] .

[0147] in, Indicates the number of sample images used in a training process. represents the i-th sample image, represents the actual attention map corresponding to the i-th sample image, represents the activation function, Represents logarithmic operation.

[0148] Optionally, the second loss function can be expressed as:

[0149] .

[0150] in, Indicates the number of sample images used in a training process. It represents the target detection image obtained by processing the i-th sample image through the target detection model. Represents the actual density map corresponding to the i-th sample image.

[0151] Optionally, the overall optimization function of the target detection model can be expressed as:

[0152] .

[0153] Among them, w1 and w2 are weight parameters, which can be preset parameters or parameters that can be adjusted during the training process. Therefore, the loss values ​​corresponding to the two branches are combined by weighted summation. During the training process, the overall loss value obtained based on the optimization function is continuously minimized to improve the target detection accuracy of the target detection model.

[0154] In an embodiment of the present application, the target detection model is adjusted based on the sample image, the actual attention map corresponding to the sample image, the actual density map corresponding to the sample image, the first loss function and the second loss function to achieve model training of the target detection model and improve the detection accuracy of the target detection model when it is applied to target detection in an image.

[0155] Corresponding to the target detection method in the above embodiment, Figure 8 This is a block diagram of the structure of the target detection device provided by the embodiment of the present disclosure. For the convenience of explanation, only the part related to the embodiment of the present disclosure is shown. Figure 8 The target detection device includes: an acquisition unit 801, an encoding unit 802, a decoding unit 803 and a generation unit 804.

[0156] An acquisition unit 801 is used to acquire an image to be detected;

[0157] The encoding unit 802 is used to encode the image through the encoding network in the target detection model to obtain the target image features of the image;

[0158] A decoding unit 803 is used to decode the target image features through the first network branch and the second network branch in the target detection model to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other through a relational modeling network.

[0159] The generating unit 804 is used to generate an object detection map of the image according to the attention map and the density map, wherein the object detection map includes a position identifier of the object in the image.

[0160] In some embodiments, there are multiple relational modeling networks, and a relational modeling network is connected between the corresponding network layers in the first network branch and the second network branch. The relational modeling network is used to establish the interaction between the image features extracted by the network layer in the first network branch and the image features extracted by the corresponding network layer in the second network branch.

[0161] In some embodiments, the decoding unit 803 is also used to: decode the input image features through the nth network layer in the first network branch to obtain the first image features; decode the input image features through the nth network layer in the second network branch to obtain the second image features; perform information aggregation processing on the first image features and the second image features through a relationship modeling network connected to the nth network layer in the first network branch and the nth network layer in the second network branch to obtain the processed first image features and the processed second image features; wherein the processed first image features are the image features of the n+1th network layer in the input first network branch, and the processed second image features are the image features of the n+1th network layer in the input second network branch, and n is greater than or equal to 1.

[0162] In some embodiments, the decoding unit 803 is also used to: perform a dot product operation on the first image feature and the second image feature in the relational modeling network to obtain an aggregate feature of the first image feature and the second image feature; based on the aggregate feature, perform information enhancement on the first image feature and the second image feature to obtain an enhanced first image feature and an enhanced second image feature; perform a dot product operation on the first image feature and the enhanced first image feature to obtain a processed first image feature, and perform a dot product operation on the second image feature and the enhanced second image feature to obtain a processed second image feature.

[0163] In some embodiments, the decoding unit 803 is also used to: in different network branches of the relational modeling network, based on the aggregated features, learn the similarity information between the first image feature and the second image feature, and obtain the similarity weight factor corresponding to the first image feature and the similarity weight factor corresponding to the second image feature; perform preliminary information enhancement on the first image feature according to the similarity weight factor corresponding to the first image feature, and obtain the preliminary enhanced first image feature; perform preliminary information enhancement on the second image feature according to the similarity weight factor corresponding to the second image feature, and obtain the preliminary enhanced second image feature; perform information enhancement on the preliminary enhanced first image feature again through a feature encoding method, and obtain the enhanced first image feature; perform information enhancement on the preliminary enhanced second image feature again through a feature encoding method, and obtain the enhanced second image feature.

[0164] In some embodiments, multiple network layers in the encoding network correspond one-to-one to multiple network layers in the first network branch and multiple network layers in the second network branch, respectively, and image features output by the network layer in the encoding network are used to input the next network layer, the corresponding network layer in the first network branch, and the corresponding network layer in the second network branch.

[0165] In some embodiments, the network layer in the encoding network adopts a residual module, and the residual module includes a Res2net network.

[0166] In some embodiments, the generating unit 804 is further used to: perform a dot product operation on the attention map and the density map to obtain the target detection map.

[0167] The target detection device provided in this embodiment can be used to execute the technical solution of the embodiment of the above-mentioned target detection method. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.

[0168] Corresponding to the model determination method of the above embodiment, Fig. 9 A structural block diagram of a model determination device provided in an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Fig. 9 The model determination device includes: an acquisition unit 901, an encoding unit 902, a decoding unit 903, a generation unit 904 and an adjustment unit 905.

[0169] An acquisition unit 901 is used to acquire a training sample, where the training sample includes a sample image, an actual attention map corresponding to the sample image, and an actual density map corresponding to the sample image;

[0170] The encoding unit 902 is used to encode the sample image through the encoding network in the target detection model to obtain the target image feature of the sample image;

[0171] A decoding unit 903 is used to decode the target image features through the first network branch and the second network branch in the target detection model to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other through a relational modeling network.

[0172] A generating unit 904, configured to generate an object detection map corresponding to the sample image according to the attention map and the density map;

[0173] The adjustment unit 905 is used to adjust the target detection model according to the first loss function, the second loss function, the sample image, the target detection map, the actual attention map and the actual density map.

[0174] The model determination device provided in this embodiment can be used to execute the technical solution of the embodiment of the above-mentioned model determination method. Its implementation principle and technical effects are similar, and this embodiment will not be repeated here.

[0175] refer to Fig.10 , which shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing the embodiment of the present disclosure, and the electronic device 1000 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0176] like Fig.10 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 to a random access memory (RAM) 1003. Various programs and data required for the operation of the electronic device 1000 are also stored in the RAM 1003. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0177] Typically, the following devices may be connected to the I / O interface 1005: input devices 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1008 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0178] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0179] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0180] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0181] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0182] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0183] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0185] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0186] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0187] In a first aspect, according to one or more embodiments of the present disclosure, a target detection method is provided, including:

[0188] Acquire an image to be detected;

[0189] In the target detection model, the image is encoded by an encoding network to obtain target image features of the image;

[0190] In the target detection model, the target image features are decoded by a first network branch and a second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network;

[0191] An object detection map of the image is generated according to the attention map and the density map, wherein the object detection map includes position identifiers of objects in the image.

[0192] According to one or more embodiments of the present disclosure, there are multiple relational modeling networks, and the relational modeling networks are connected between the first network branches and the corresponding network layers in the second network branches. The relational modeling networks are used to establish interaction between image features extracted by the network layer in the first network branch and image features extracted by the corresponding network layer in the second network branch.

[0193] According to one or more embodiments of the present disclosure, the decoding process of the target image feature by the first network branch and the second network branch includes:

[0194] Decoding the input image feature through the nth network layer in the first network branch to obtain a first image feature;

[0195] Decoding the input image feature through the nth network layer in the second network branch to obtain a second image feature;

[0196] performing information aggregation processing on the first image feature and the second image feature through a relational modeling network connected to the nth network layer in the first network branch and the nth network layer in the second network branch to obtain a processed first image feature and a processed second image feature;

[0197] Among them, the processed first image feature is the image feature of the n+1th network layer input in the first network branch, and the processed second image feature is the image feature of the n+1th network layer input in the second network branch, where n is greater than or equal to 1.

[0198] According to one or more embodiments of the present disclosure, the information aggregation processing is performed on the first image feature and the second image feature through the relationship modeling network connected to the nth network layer in the first network branch and the nth network layer in the second network branch to obtain the processed first image feature and the processed second image feature, including:

[0199] In the relational modeling network, performing a dot product operation on the first image feature and the second image feature to obtain an aggregated feature of the first image feature and the second image feature;

[0200] Based on the aggregated features, respectively enhance the first image features and the second image features to obtain enhanced first image features and enhanced second image features;

[0201] A dot product operation is performed on the first image feature and the enhanced first image feature to obtain the processed first image feature, and a dot product operation is performed on the second image feature and the enhanced second image feature to obtain the processed second image feature.

[0202] According to one or more embodiments of the present disclosure, the information enhancement of the first image feature and the second image feature is respectively performed based on the aggregated feature to obtain the enhanced first image feature and the enhanced second image feature, including:

[0203] In different network branches of the relationship modeling network, based on the aggregated features, similarity information between the first image feature and the second image feature is learned to obtain a similarity weight factor corresponding to the first image feature and a similarity weight factor corresponding to the second image feature;

[0204] Performing preliminary information enhancement on the first image feature according to the similarity weight factor corresponding to the first image feature to obtain a preliminary enhanced first image feature;

[0205] Performing preliminary information enhancement on the second image feature according to the similarity weight factor corresponding to the second image feature to obtain a preliminary enhanced second image feature;

[0206] By means of feature encoding, the information of the initially enhanced first image feature is further enhanced to obtain an enhanced first image feature;

[0207] The information of the preliminarily enhanced second image feature is enhanced again by feature encoding to obtain an enhanced second image feature.

[0208] According to one or more embodiments of the present disclosure, the multiple network layers in the encoding network correspond one-to-one to the multiple network layers in the first network branch and the multiple network layers in the second network branch, respectively, and the image features output by the network layers in the encoding network are used to input the next network layer, the corresponding network layer in the first network branch, and the corresponding network layer in the second network branch.

[0209] According to one or more embodiments of the present disclosure, there are multiple relationship modeling networks, and the first network branch and the second network branch use residual modules for the network layers in the encoding network one by one, and the residual modules include a Res2net network.

[0210] According to one or more embodiments of the present disclosure, generating an object detection map of the image according to the attention map and the density map includes:

[0211] A dot product operation is performed on the attention map and the density map to obtain the target detection map.

[0212] In a second aspect, according to one or more embodiments of the present disclosure, a model determination method is provided, including:

[0213] Acquire a training sample, wherein the training sample includes a sample image, an actual attention map corresponding to the sample image, and an actual density map corresponding to the sample image;

[0214] In the target detection model, the sample image is encoded by a coding network to obtain target image features of the sample image;

[0215] In the target detection model, the target image features are decoded by a first network branch and a second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network;

[0216] Generating a target detection map corresponding to the sample image according to the attention map and the density map;

[0217] The target detection model is adjusted according to the first loss function, the second loss function, the sample image, the target detection map, the actual attention map and the actual density map.

[0218] In a third aspect, according to one or more embodiments of the present disclosure, there is provided a target detection device, including:

[0219] An acquisition unit, used for acquiring an image to be detected;

[0220] An encoding unit, used to encode the image through an encoding network in a target detection model to obtain target image features of the image;

[0221] A decoding unit, configured to decode the target image features through a first network branch and a second network branch in the target detection model to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network;

[0222] A generating unit is used to generate a target detection map of the image according to the attention map and the density map, wherein the target detection map includes a position identifier of the target in the image.

[0223] According to one or more embodiments of the present disclosure, there are multiple relational modeling networks, and the relational modeling networks are connected between the first network branches and the corresponding network layers in the second network branches. The relational modeling networks are used to establish interaction between image features extracted by the network layer in the first network branch and image features extracted by the corresponding network layer in the second network branch.

[0224] According to one or more embodiments of the present disclosure, the decoding unit is further configured to

[0225] Decoding the input image feature through the nth network layer in the first network branch to obtain a first image feature;

[0226] Decoding the input image feature through the nth network layer in the second network branch to obtain a second image feature;

[0227] performing information aggregation processing on the first image feature and the second image feature through a relational modeling network connected to the nth network layer in the first network branch and the nth network layer in the second network branch to obtain a processed first image feature and a processed second image feature;

[0228] Among them, the processed first image feature is the image feature of the n+1th network layer input in the first network branch, and the processed second image feature is the image feature of the n+1th network layer input in the second network branch, where n is greater than or equal to 1.

[0229] According to one or more embodiments of the present disclosure, the decoding unit is further configured to:

[0230] In the relational modeling network, performing a dot product operation on the first image feature and the second image feature to obtain an aggregated feature of the first image feature and the second image feature;

[0231] Based on the aggregated features, respectively enhance the first image features and the second image features to obtain enhanced first image features and enhanced second image features;

[0232] A dot product operation is performed on the first image feature and the enhanced first image feature to obtain the processed first image feature, and a dot product operation is performed on the second image feature and the enhanced second image feature to obtain the processed second image feature.

[0233] According to one or more embodiments of the present disclosure, the decoding unit is further configured to:

[0234] In different network branches of the relationship modeling network, based on the aggregated features, similarity information between the first image feature and the second image feature is learned to obtain a similarity weight factor corresponding to the first image feature and a similarity weight factor corresponding to the second image feature;

[0235] Performing preliminary information enhancement on the first image feature according to the similarity weight factor corresponding to the first image feature to obtain a preliminary enhanced first image feature;

[0236] Performing preliminary information enhancement on the second image feature according to the similarity weight factor corresponding to the second image feature to obtain a preliminary enhanced second image feature;

[0237] By means of feature encoding, the information of the initially enhanced first image feature is further enhanced to obtain an enhanced first image feature;

[0238] The information of the preliminarily enhanced second image feature is enhanced again by feature encoding to obtain an enhanced second image feature.

[0239] According to one or more embodiments of the present disclosure, the multiple network layers in the encoding network correspond one-to-one to the multiple network layers in the first network branch and the multiple network layers in the second network branch, respectively, and the image features output by the network layers in the encoding network are used to input the next network layer, the corresponding network layer in the first network branch, and the corresponding network layer in the second network branch.

[0240] According to one or more embodiments of the present disclosure, the network layer in the encoding network adopts a residual module, and the residual module includes a Res2net network.

[0241] According to one or more embodiments of the present disclosure, the generating unit is further configured to:

[0242] A dot product operation is performed on the attention map and the density map to obtain the target detection map.

[0243] In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided an image determination device, including:

[0244] An acquisition unit, configured to acquire a training sample, wherein the training sample includes a sample image, an actual attention map corresponding to the sample image, and an actual density map corresponding to the sample image;

[0245] An encoding unit, used to encode the sample image through an encoding network in a target detection model to obtain a target image feature of the sample image;

[0246] A decoding unit, configured to decode the target image features through a first network branch and a second network branch in the target detection model to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other in terms of image features through a relational modeling network;

[0247] A generating unit, configured to generate a target detection map corresponding to the sample image according to the attention map and the density map;

[0248] An adjustment unit is used to adjust the target detection model according to the first loss function, the second loss function, the sample image, the target detection map, the actual attention map and the actual density map.

[0249] In a fifth aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0250] The memory stores computer-executable instructions;

[0251] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the target detection method described in the first aspect or various possible designs of the first aspect, or executes the model determination method described in the second aspect or various possible designs of the second aspect.

[0252] In a sixth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the target detection method described in the first aspect and various possible designs of the first aspect is implemented, or the model determination method described in the second aspect or various possible designs of the second aspect is implemented.

[0253] In the seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, wherein the computer program product includes computer execution instructions. When a processor executes the computer execution instructions, the target detection method as described in the first aspect and various possible designs of the first aspect is implemented, or the model determination method as described in the second aspect or various possible designs of the second aspect is implemented.

[0254] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0255] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0256] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A target detection method, comprising: Acquire an image to be detected; In the target detection model, the image is encoded by an encoding network to obtain target image features of the image; In the target detection model, the target image features are decoded by the first network branch and the second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other through a relational modeling network, and the interaction process includes: adjusting the network features extracted by the second network branch based on the image features extracted by the first network branch, and feeding back the adjusted network features to the second network branch; adjusting the network features extracted by the first network branch based on the image features extracted by the second network branch, and feeding back the adjusted network features to the first network branch; An object detection map of the image is generated according to the attention map and the density map, wherein the object detection map includes position identifiers of objects in the image.

2. According to the target detection method according to claim 1, there are multiple relationship modeling networks, and the relationship modeling network is connected between the one-to-one corresponding network layers in the first network branch and the second network branch, and the relationship modeling network is used to establish the interaction between the image features extracted by the network layer in the first network branch and the image features extracted by the corresponding network layer in the second network branch.

3. According to the target detection method of claim 2, the decoding process of the target image features by the first network branch and the second network branch comprises: Decoding the input image feature through the nth network layer in the first network branch to obtain a first image feature; Decoding the input image feature through the nth network layer in the second network branch to obtain a second image feature; performing information aggregation processing on the first image feature and the second image feature through a relational modeling network connected to the nth network layer in the first network branch and the nth network layer in the second network branch to obtain a processed first image feature and a processed second image feature; Among them, the processed first image feature is the image feature of the n+1th network layer input in the first network branch, and the processed second image feature is the image feature of the n+1th network layer input in the second network branch, where n is greater than or equal to 1.

4. The target detection method according to claim 3, wherein the information aggregation processing is performed on the first image feature and the second image feature through the relationship modeling network connected to the nth network layer in the first network branch and the nth network layer in the second network branch to obtain the processed first image feature and the processed second image feature, comprising: In the relational modeling network, performing a dot product operation on the first image feature and the second image feature to obtain an aggregated feature of the first image feature and the second image feature; Based on the aggregated features, respectively enhance the first image features and the second image features to obtain enhanced first image features and enhanced second image features; A dot product operation is performed on the first image feature and the enhanced first image feature to obtain the processed first image feature, and a dot product operation is performed on the second image feature and the enhanced second image feature to obtain the processed second image feature.

5. The target detection method according to claim 4, wherein based on the aggregated features, the first image features and the second image features are respectively enhanced to obtain enhanced first image features and enhanced second image features, comprising: In different network branches of the relationship modeling network, based on the aggregated features, similarity information between the first image feature and the second image feature is learned to obtain a similarity weight factor corresponding to the first image feature and a similarity weight factor corresponding to the second image feature; Performing preliminary information enhancement on the first image feature according to the similarity weight factor corresponding to the first image feature to obtain a preliminary enhanced first image feature; Performing preliminary information enhancement on the second image feature according to the similarity weight factor corresponding to the second image feature to obtain a preliminary enhanced second image feature; By means of feature encoding, the information of the initially enhanced first image feature is further enhanced to obtain an enhanced first image feature; The information of the preliminarily enhanced second image feature is enhanced again by feature encoding to obtain an enhanced second image feature.

6. According to the target detection method described in any one of claims 1 to 5, the multiple network layers in the encoding network correspond one-to-one to the multiple network layers in the first network branch and the multiple network layers in the second network branch, respectively, and the image features output by the network layers in the encoding network are used to input the next network layer, the corresponding network layer in the first network branch, and the corresponding network layer in the second network branch.

7. According to the target detection method according to any one of claims 1-5, the network layer in the encoding network adopts a residual module, and the residual module includes a Res2net network.

8. The target detection method according to any one of claims 1 to 5, wherein generating the target detection map of the image according to the attention map and the density map comprises: A dot product operation is performed on the attention map and the density map to obtain the target detection map.

9. A model determination method, comprising: Acquire a training sample, wherein the training sample includes a sample image, an actual attention map corresponding to the sample image, and an actual density map corresponding to the sample image; In the target detection model, the sample image is encoded by a coding network to obtain target image features of the sample image; In the target detection model, the target image features are decoded by the first network branch and the second network branch to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other through a relational modeling network, and the interaction process includes: adjusting the network features extracted by the second network branch based on the image features extracted by the first network branch, and feeding back the adjusted network features to the second network branch; adjusting the network features extracted by the first network branch based on the image features extracted by the second network branch, and feeding back the adjusted network features to the first network branch; Generating a target detection map corresponding to the sample image according to the attention map and the density map; The target detection model is adjusted according to the first loss function, the second loss function, the sample image, the target detection map, the actual attention map and the actual density map.

10. A target detection device, comprising: An acquisition unit, used for acquiring an image to be detected; An encoding unit, used to encode the image through an encoding network in a target detection model to obtain target image features of the image; A decoding unit, used for decoding the target image features through the first network branch and the second network branch in the target detection model to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other through a relational modeling network, and the interaction process includes: adjusting the network features extracted by the second network branch based on the image features extracted by the first network branch, and feeding back the adjusted network features to the second network branch; adjusting the network features extracted by the first network branch based on the image features extracted by the second network branch, and feeding back the adjusted network features to the first network branch; A generating unit is used to generate a target detection map of the image according to the attention map and the density map, wherein the target detection map includes a position identifier of the target in the image.

11. An image determination device, comprising: An acquisition unit, configured to acquire a training sample, wherein the training sample includes a sample image, an actual attention map corresponding to the sample image, and an actual density map corresponding to the sample image; An encoding unit, used to encode the sample image through an encoding network in a target detection model to obtain a target image feature of the sample image; A decoding unit, used for decoding the target image features through the first network branch and the second network branch in the target detection model to obtain an attention map output by the first network branch and a density map output by the second network branch, wherein the first network branch and the second network branch interact with each other through a relational modeling network, and the interaction process includes: adjusting the network features extracted by the second network branch based on the image features extracted by the first network branch, and feeding back the adjusted network features to the second network branch; adjusting the network features extracted by the first network branch based on the image features extracted by the second network branch, and feeding back the adjusted network features to the first network branch; A generating unit, configured to generate a target detection map corresponding to the sample image according to the attention map and the density map; An adjustment unit is used to adjust the target detection model according to the first loss function, the second loss function, the sample image, the target detection map, the actual attention map and the actual density map.

12. An electronic device comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the target detection method according to any one of claims 1 to 8 or the model determination method according to claim 9.

13. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the target detection method according to any one of claims 1 to 8 or the model determination method according to claim 9 is implemented.

14. A computer program product, comprising computer executable instructions, and when a processor executes the computer executable instructions, the target detection method according to any one of claims 1 to 8 or the model determination method according to claim 9 is implemented.

Images