Target detection method and device based on fine-grained feature extraction

Abstract

The present invention relates to the technical field of computer vision and target detection, and provides a target detection method and device based on fine-grained feature extraction. The method inputs an image to be detected into a detection model and obtains a detection result of the target output by the detection model. The detection model includes a feature extraction module, a fine-grained feature extraction and fusion module, and a detector connected in sequence. The feature extraction module extracts multiple image features of different resolutions of the image to be detected. The fine-grained feature extraction and fusion module associates a first image feature with other image features among the multiple image features of different resolutions based on an attention mechanism, extracts target fine-grained features based on the association results, fuses the target fine-grained features with the first image features, and splices the obtained fusion results with the second image features in the channel dimension, so that the detector uses the splicing results to obtain a detection result. The detection result is more accurate and has high detection accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and target detection, and in particular to a target detection method and device based on fine-grained feature extraction. Background Art

[0002] Computer vision has always been a hot topic and a difficult topic in computer science research, and object detection has become a very important research issue in the field of computer vision.

[0003] Existing target detection technologies usually use convolutional neural network models to detect target images. However, the feature information of small targets and severely occluded targets is difficult to be extracted by convolutional neural network models, resulting in missed detections and false detections.

[0004] Therefore, there is an urgent need to provide a target detection method. Summary of the Invention

[0005] The present invention provides a target detection method and device based on fine-grained feature extraction, which are used to solve the defects in the prior art.

[0006] The present invention provides a target detection method based on fine-grained feature extraction, comprising:

[0007] Obtain the image of the target to be detected;

[0008] Inputting the image to be detected into a detection model to obtain a detection result of the target output by the detection model;

[0009] The detection model includes a feature extraction module, a fine-grained feature extraction and fusion module, and a detector connected in sequence; the detection model is trained based on sample images carrying sample labels;

[0010] The feature extraction module is used to extract image features of multiple different resolutions of the image to be detected;

[0011] The fine-grained feature extraction and fusion module is configured to associate a first image feature with other image features among the multiple image features of different resolutions based on an attention mechanism, extract a target fine-grained feature based on the obtained association result, fuse the target fine-grained feature with the first image feature, and splice the obtained fusion result with the second image feature among the multiple image features of different resolutions in the channel dimension to obtain a splicing result;

[0012] The detector is used to obtain the detection result based on the splicing result.

[0013] According to a target detection method based on fine-grained feature extraction provided by the present invention, the fine-grained feature extraction and fusion module is specifically used for:

[0014] Projecting the plurality of image features at different resolutions into a coding space, and calculating, in the coding space, an attention map for characterizing the correlation between the first image feature and the other image features;

[0015] Calculate the feature representations of all positions in the other image features, and perform weighted summation of the feature representations and the attention map to obtain the target fine-grained features.

[0016] According to a target detection method based on fine-grained feature extraction provided by the present invention, the first image feature is the image feature with the lowest resolution among the multiple image features with different resolutions; and / or,

[0017] The second image feature is an image feature with the highest resolution among the multiple image features with different resolutions.

[0018] According to a target detection method based on fine-grained feature extraction provided by the present invention, the number of image features is 4;

[0019] The feature extraction module is HRNet which includes four feature extraction stages;

[0020] The image to be detected passes through the four feature extraction stages in sequence, and the fourth feature extraction stage outputs the image features.

[0021] According to a target detection method based on fine-grained feature extraction provided by the present invention, the detection model is obtained by performing distributed training on an initial model based on the sample image.

[0022] According to a target detection method based on fine-grained feature extraction provided by the present invention, the detector includes a dimensionality reduction module, a target center point prediction module, a target scale prediction module, and a target center point offset prediction module, wherein the dimensionality reduction module is connected to the target center point prediction module, the target scale prediction module, and the target center point offset prediction module respectively;

[0023] The dimensionality reduction module is used to perform dimensionality reduction processing on the splicing result to obtain a dimensionality reduction result;

[0024] The target center point prediction module is used to determine the target center point based on the dimensionality reduction result; the target scale prediction module is used to determine the target scale based on the dimensionality reduction result; and the target center point offset prediction module is used to determine the target center point offset based on the dimensionality reduction result.

[0025] According to a target detection method based on fine-grained feature extraction provided by the present invention, the detection result includes a target bounding box;

[0026] Accordingly, the detector is also used for:

[0027] Generate a target candidate frame based on the target center point and the target scale;

[0028] Based on the target center point offset, the target candidate box is fine-tuned to obtain the target bounding box.

[0029] The present invention also provides a target detection device based on fine-grained feature extraction, comprising:

[0030] An image acquisition module is used to acquire an image of the target to be detected;

[0031] The target detection module is used to input the image to be detected into the detection model and obtain the detection result of the target output by the detection model;

[0032] The detection model includes a feature extraction module, a fine-grained feature extraction and fusion module, and a detector connected in sequence; the detection model is trained based on sample images carrying sample labels;

[0033] The feature extraction module is used to extract image features of multiple different resolutions of the image to be detected;

[0034] The fine-grained feature extraction and fusion module is configured to associate a first image feature with other image features among the multiple image features of different resolutions based on an attention mechanism, extract a target fine-grained feature based on the obtained association result, fuse the target fine-grained feature with the first image feature, and splice the obtained fusion result with the second image feature among the multiple image features of different resolutions in the channel dimension to obtain a splicing result;

[0035] The detector is used to obtain the detection result based on the splicing result.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the target detection method based on fine-grained feature extraction as described above is implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described target detection methods based on fine-grained feature extraction.

[0038] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described target detection methods based on fine-grained feature extraction.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention provides a target detection method and device based on fine-grained feature extraction. The method first obtains an image of the target to be detected; then inputs the image to be detected into a detection model to obtain a detection result of the target output by the detection model. The detection model may include a feature extraction module, a fine-grained feature extraction and fusion module and a detector connected in sequence. The feature extraction module extracts multiple image features of different resolutions of the image to be detected, and higher-level semantic information can be obtained without losing spatial resolution, greatly improving the detection model's ability to describe the target; the fine-grained feature extraction and fusion module associates the first image feature with other image features in multiple image features of different resolutions based on the attention mechanism, and extracts the target fine-grained features based on the obtained association results, which can achieve global long-distance dependency across feature domains, fuse the target fine-grained features and the first image features, and splice the obtained fusion results with the second image features in the channel dimension, which can prevent the loss of target position information in the process of extracting fine-grained features, improve the performance of the detection model, and improve the detection accuracy of the detection model for occluded targets, thereby making the detection results obtained by the detector using the splicing results more accurate, which can improve the detection accuracy of the detection model for occluded targets and reduce the possibility of missed detection and false detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without any creative work.

[0042] Figure 1 This is one of the flow charts of the target detection method based on fine-grained feature extraction provided by the present invention;

[0043] Figure 2 Schematic diagram of the structure of the detection model in the target detection method based on fine-grained feature extraction provided by the present invention;

[0044] Figure 3 Schematic diagram of the structure of the fine-grained feature extraction and fusion module of the detection model in the target detection method based on fine-grained feature extraction provided by the present invention;

[0045] Figure 4 Schematic diagram of the structure of the feature extraction module of the detection model in the target detection method based on fine-grained feature extraction provided by the present invention;

[0046] Figure 5 Schematic diagram of the structure of the target detection device based on fine-grained feature extraction provided by the present invention;

[0047] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0049] Existing target detection technologies typically use convolutional neural network models to detect target images, but they do not extract fine-grained features of the target. Furthermore, convolutional neural network models can only learn information from local regions, making it difficult to accurately extract feature information for small and heavily occluded targets, leading to missed detections and false detections. Therefore, an embodiment of the present invention provides a target detection method based on fine-grained feature extraction to address the shortcomings of existing target detection technologies.

[0050] Figure 1 FIG. 1 is a flow chart of a target detection method based on fine-grained feature extraction provided in an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0051] S1, obtain the image of the target to be detected;

[0052] S2, inputting the image to be detected into a detection model to obtain a detection result of the target output by the detection model;

[0053] The detection model includes a feature extraction module, a fine-grained feature extraction and fusion module, and a detector connected in sequence; the detection model is trained based on sample images carrying sample labels;

[0054] The feature extraction module is used to extract image features of multiple different resolutions of the image to be detected;

[0055] The fine-grained feature extraction and fusion module is configured to associate a first image feature with other image features among the multiple image features of different resolutions based on an attention mechanism, extract a target fine-grained feature based on the obtained association result, fuse the target fine-grained feature with the first image feature, and splice the obtained fusion result with the second image feature among the multiple image features of different resolutions in the channel dimension to obtain a splicing result;

[0056] The detector is used to obtain the detection result based on the splicing result.

[0057] Specifically, the target detection method based on fine-grained feature extraction provided in the embodiment of the present invention is executed by a target detection device based on fine-grained feature extraction. The device can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0058] First, step S1 is executed to acquire an image of the target to be detected. The target can be a pedestrian, a manufactured product or equipment, or an everyday object. The image to be detected can be captured by an image acquisition device, which can be a standard camera, an industrial camera, or the like, and is not specifically limited here. The image to be detected can have a height of H, a width of W, and a channel count of 3.

[0059] Then, step S2 is executed to introduce the detection model, input the image to be detected into the detection model, and obtain the detection result of the target output by the detection model.

[0060] like Figure 2 As shown, the detection model can include a feature extraction module, a fine-grained feature extraction and fusion module, and a detector connected in sequence. The image to be detected is sequentially processed by the feature extraction module for feature extraction, the fine-grained feature extraction and fusion module for target fine-grained feature extraction and fusion, and the detector for target detection, ultimately obtaining and outputting the detection result.

[0061] The detection result may include the target center point, target scale, and target center offset in the image to be detected, or it may simply include the target bounding box in the image to be detected. It is understood that the target center point is the center position of the target in the image to be detected, and the target center offset is used to represent the degree of offset between the target center point and the actual center point. The target scale may include information such as the width and height of the target in the image to be detected. The target bounding box can be generated using the target center point, target scale, and target center offset.

[0062] The feature extraction module can be a backbone network comprising multiple branches, each for extracting image features at different resolutions from the image to be detected. Image features at different resolutions, i.e., at different scales, can be represented using feature maps. Image features correspond one-to-one to branches of the feature extraction model, and the number of branches can be set as needed, for example, to four or another number, without specific limitation.

[0063] Since this feature extraction module can extract image features of multiple different resolutions, it can obtain higher-level semantic information without losing spatial resolution, greatly improving the detection model's ability to describe the target.

[0064] The fine-grained feature extraction and fusion module can use the attention mechanism to associate the first image feature with other image features among multiple image features of different resolutions to obtain an association result. The first image feature does not specifically refer to the first image feature, but can be any image feature. It is only used here to distinguish it from other image features. The number of other image features can be one or more. When there are multiple other image features, the first image feature is associated with each other image feature to obtain multiple association results. For example, if the number of image features is 4, the number of other image features is 3, and thus 3 association results can be obtained.

[0065] It is understood that the association can be achieved by calculating the correlation between the first image feature and other image features through a function, thereby obtaining a correlation result. The correlation result can be an attention map, which contains the correlation between the first image feature and the fine-grained information of all targets in the other image features.

[0066] The association results can then be used to determine the target fine-grained features. The target fine-grained features corresponding to each association result can be obtained by weighted summing the association result with the corresponding other image features. The target fine-grained features corresponding to each association result contain the fine-grained information of all targets.

[0067] Afterwards, the target fine-grained features and the first image features can be fused to produce a fusion result. Since the target fine-grained features corresponding to each association result contain different fine-grained information, a fusion method can be employed to first superimpose the target fine-grained features corresponding to each association result to produce a superimposed result. This fusion result is a richer target fine-grained feature. Furthermore, the superimposed result can be fused with the first image features using other fusion methods, which are not specifically limited here.

[0068] Furthermore, the fusion result can be concatenated with the second image feature in the channel dimension to obtain a concatenated result. The second image feature can be any other image feature different from the first image feature. Since the resolutions of the first and second image features differ, the resolutions of the fusion result and the second image feature also differ. Therefore, the fusion result can be sampled to obtain a sampled result, so that the sampled result has the same resolution as the second image feature. This sampled result can then be used to concatenate with the second image feature in the channel dimension.

[0069] The fine-grained feature extraction and fusion module uses the attention mechanism to associate the first image feature with other image features in multiple image features of different resolutions to obtain an association result, and extracts the target fine-grained features based on the obtained association result. It can establish long-distance dependencies globally across feature domains, and at the same time fuse the target fine-grained features with the first image features, and splice the fusion results with the second image features in the channel dimension, which can prevent the loss of target position information in the process of extracting fine-grained features and improve the detection accuracy of the detection model for occluded targets.

[0070] Finally, the detector is used to obtain the detection result of the target in the image to be detected by using the stitching result. The structure of the detector can be set according to the content of the required detection result and is not specifically limited here.

[0071] The detection model can be obtained by training the initial model using sample images carrying sample labels. The sample labels can be sample bounding boxes annotated in the sample images. The fine-grained feature extraction and fusion module does not require additional fine-grained information annotation.

[0072] When training the initial model, the sample image can be first input into the initial model to obtain the detection result output by the initial model, and then the loss function value is calculated based on the detection result and the sample label. Finally, the model parameters of the initial model are updated according to the loss function value; the above input process and calculation process are iteratively executed until the loss function converges or the preset number of iterations is reached to obtain the detection model.

[0073] The target detection method based on fine-grained feature extraction provided in an embodiment of the present invention first obtains an image of the target to be detected; then inputs the image to be detected into a detection model to obtain a detection result of the target output by the detection model. The detection model may include a feature extraction module, a fine-grained feature extraction and fusion module and a detector connected in sequence. The feature extraction module extracts multiple image features of different resolutions of the image to be detected, and higher-level semantic information can be obtained without losing spatial resolution, greatly improving the detection model's ability to describe the target; the fine-grained feature extraction and fusion module associates the first image feature with other image features in multiple image features of different resolutions based on the attention mechanism, and extracts the target fine-grained features based on the obtained association results, which can achieve global long-distance dependency across feature domains, fuse the target fine-grained features and the first image features, and splice the fusion results with the second image features in the channel dimension, which can prevent the loss of target position information in the process of extracting fine-grained features, improve the performance of the detection model, and improve the detection accuracy of the detection model for occluded targets, thereby making the detection results obtained by the detector using the splicing results more accurate, which can improve the detection accuracy of the detection model for occluded targets and reduce the possibility of missed detection and false detection.

[0074] On the basis of the above embodiments, in the object detection method based on fine-grained feature extraction provided in the embodiments of the present invention, the fine-grained feature extraction and fusion module is specifically used for:

[0075] Projecting the plurality of image features at different resolutions into a coding space, and calculating, in the coding space, an attention map for characterizing the correlation between the first image feature and the other image features;

[0076] Calculate the feature representations of all positions in the other image features, and perform weighted summation of the feature representations and the attention map to obtain the target fine-grained features.

[0077] Specifically, in an embodiment of the present invention, the fine-grained feature extraction and fusion module can first project multiple image features of different resolutions into the coding space, which can be implemented using a 1×1 convolution kernel. Here, the first image feature can be used as a query feature, and other image features can be used as response features. By associating the query feature with the response feature, more discriminative fine-grained features of the target can be learned. The first image feature and other image features can use different 1×1 convolution kernels. For example, the 1×1 convolution kernel used for the first image feature can be transformed linearly. Characterization, the 1×1 convolution kernel used by other image features can be linearly transformed representation.

[0078] Afterwards, in the encoding space, the pairwise function can be used Calculate the attention map, a and b both represent pairwise functions The pairwise function This can be achieved through dot product. After taking the value of , the pairwise function can be normalized by the exponential function (i.e., the softmax function) The values ​​are mapped to [0,1], and all pairwise functions are guaranteed to be The sum of the values ​​is 1. The pairwise function after mapping The value of can be used as the attention map, which can improve the accuracy of the attention map to highlight the required information.

[0079] The attention map is the association result, and each pixel value in the attention map represents an attention weight, which is used to characterize the correlation between the first image feature and other image features.

[0080] Simultaneously, the feature representations of all positions in other image features can be calculated, and the feature representations of all positions can be calculated using a 1×1 convolution kernel, which can be obtained by a learnable linear transformation g To characterize.

[0081] Afterwards, the feature representations of all positions in the other image features can be weighted and summed with the attention map to obtain the target fine-grained feature, which can be represented by a feature map containing the target fine-grained information. Here, the target fine-grained feature can be transformed into a channel dimension through a 1×1 convolution kernel so that the target fine-grained feature has the same channel dimension as the first image feature, so that the two can be smoothly fused.

[0082] Finally, the target fine-grained features and the first image features can be fused by adding them together to obtain a fusion result. Here, it can be understood that a residual branch is added between the target fine-grained features and the first image features, which can improve the performance and stability of the detection model.

[0083] If there are two image features, then the other image features are one. In this case, the first image feature can be expressed as , other image features can be expressed as , the fusion result can be expressed as , and there is:

[0084] (1)

[0085] (2)

[0086] in, It is a 1×1 convolution kernel that keeps the channel dimension of the target fine-grained feature the same as that of the first image feature. is the projection result of the first image feature in the encoding space, is the projection result of other image features in the encoding space, is the attention map corresponding to other image features, is the feature representation of all positions in other image features, 、 and represent the height, width and channel dimensions of the first image feature, respectively, 、 and Represent the height, width, and channel dimensions of other image features respectively.

[0087] If there are 4 image features, then there are 3 other image features. Figure 3 As shown, the i-th other image feature can be expressed as , and there is:

[0088] (3)

[0089] The fusion result can be expressed as , and there is:

[0090] (4)

[0091] in, It is a 1×1 convolution kernel that keeps the channel dimension of the target fine-grained feature the same as that of the first image feature. is the i-th other image feature The corresponding target fine-grained feature with the same channel dimension as the first image feature, is the projection result of the first image feature in the encoding space, is the projection result of the i-th other image feature in the encoding space, is the attention map corresponding to the i-th other image feature, is the feature representation of all positions in the i-th other image feature, 、 and represent the height, width and channel dimensions of the first image feature, respectively, 、 and Represent the height, width and channel dimensions of the i-th other image feature respectively.

[0092] It is understandable that Figure 3 In As the first image feature, 、 and As other image features, and As the second image feature, “×” represents multiplication and “+” represents addition. is the first other image feature Corresponding to the first image feature The target fine-grained features with the same channel dimension, is the second other image feature Corresponding to the first image feature The target fine-grained features with the same channel dimension, is the third other image feature Corresponding to the first image feature The target fine-grained features have the same channel dimension.

[0093] In an embodiment of the present invention, by calculating the attention map in the coding space and fusing the obtained fine-grained target features and the first image features by weighted summing the feature representations of all positions in other image features with the attention map, it is possible to establish long-distance dependencies globally across feature domains, so that the obtained fusion results provide a theoretical basis for improving the detection accuracy of the detection model for occluded targets.

[0094] On the basis of the above embodiment, in the target detection method based on fine-grained feature extraction provided in an embodiment of the present invention, the first image feature is the image feature with the lowest resolution among the multiple image features with different resolutions; and / or,

[0095] The second image feature is an image feature with the highest resolution among the multiple image features with different resolutions.

[0096] Specifically, in the embodiment of the present invention, in order to reduce the computational overhead of the detection model, the image feature with the lowest resolution among multiple image features with different resolutions can be used as the first image feature, or the image feature with the highest resolution among multiple image features with different resolutions can be used as the second image feature. In this case, the fusion result needs to be upsampled so that the obtained sampling result has the same resolution as the second image feature. Figure 3 The sampling in is upsampling.

[0097] On the basis of the above embodiment, in the object detection method based on fine-grained feature extraction provided in the embodiment of the present invention, the number of the image features is 4;

[0098] The feature extraction module is HRNet which includes four feature extraction stages;

[0099] The image to be detected passes through the four feature extraction stages in sequence, and the fourth feature extraction stage outputs the image features.

[0100] Specifically, in the embodiment of the present invention, the number of image features may be 4, and the feature extraction module for extracting image features may be HRNet including 4 feature extraction stages. Figure 4 As shown in the figure, the four feature extraction stages are stage 1, stage 2, stage 3, and stage 4. HRNet has a parallel structure and includes four extraction branches with different resolutions. The extraction branches can be divided into extraction branch 1, extraction branch 2, extraction branch 3, and extraction branch 4 in descending order of resolution. Each extraction branch has corresponding sub-branches in each feature extraction stage. Figure 4 Each row is an extraction branch. The top row corresponds to extraction branch 1, and the height of its feature map is H / 4 and the width is W / 4; the second row corresponds to extraction branch 2, and the height of its feature map is H / 8 and the width is W / 8; the third row corresponds to extraction branch 3, and the height of its feature map is H / 16 and the width is W / 16; the bottom row corresponds to extraction branch 4, and the height of its feature map is H / 32 and the width is W / 32.

[0101] Starting from stage 1, feature extraction is performed on the image to be inspected using extraction branch 1, and extraction branches are gradually added in parallel, from high resolution to low resolution. Specifically, stage 1 contains only one highest resolution branch, extraction branch 1. From stage 2 to stage 4, a parallel branch is added in each stage, and the output of each branch in the previous stage is used as the input of each branch in the next stage. The resolution of the new branch in each stage is half that of the lowest resolution branch in the previous stage, and the number of channels is doubled.

[0102] The feature maps of different resolutions between parallel branches in each stage have different granularity, focusing on regions of different scales in the image to be detected. By upsampling or downsampling feature maps of different resolutions, the feature maps from different branches are fused, enabling information exchange between representations of different resolutions. This improves the expressive power of high-resolution and low-resolution feature maps, allowing multi-resolution representations to better complement each other.

[0103] Finally, four feature maps with different resolutions are output, that is, four image features with different resolutions, which are represented as , whose resolutions are respectively , the channel dimensions are 32, 64, 128, and 256 respectively. HRNet can efficiently extract the high-level semantic features (i.e., image features) of the target while preserving the location information of the target.

[0104] Figure 4In the figure, horizontal arrows are used to represent convolutional feature extraction, downward arrows are used to represent downsampling, and upward arrows are used to represent upsampling.

[0105] In an embodiment of the present invention, HRNet, which includes four feature extraction stages, is used as a feature extraction module. It can always retain high-resolution features while extracting image feature information, thereby obtaining higher-level semantic information without losing spatial resolution, greatly improving the detection model's ability to describe the target.

[0106] On the basis of the above-mentioned embodiment, in the target detection method based on fine-grained feature extraction provided in the embodiment of the present invention, the detection model is obtained by performing distributed training on the initial model based on the sample image.

[0107] Specifically, in an embodiment of the present invention, since the detection model may include a feature extraction module, a fine-grained feature extraction and fusion module, and a detector, a distributed training method may be used to train the initial model to obtain a detection model to improve training efficiency.

[0108] Here, the distributed training method refers to a parallel training method, which may include a training method based on PyTorch implementation or a parallel training method based on other deep learning frameworks, and is not specifically limited here.

[0109] On the basis of the above embodiment, the object detection method based on fine-grained feature extraction provided in the embodiment of the present invention includes a dimensionality reduction module, a target center point prediction module, a target scale prediction module, and a target center point offset prediction module, wherein the dimensionality reduction module is connected to the target center point prediction module, the target scale prediction module, and the target center point offset prediction module respectively;

[0110] The dimensionality reduction module is used to perform dimensionality reduction processing on the splicing result to obtain a dimensionality reduction result;

[0111] The target center prediction module is used to determine the target center based on the dimensionality reduction results;

[0112] The target scale prediction module is used to determine the target scale based on the dimensionality reduction result;

[0113] The target center point offset prediction module is used to determine the target center point offset based on the dimensionality reduction result.

[0114] Specifically, the detector can be a center point-based detector, the dimensionality reduction module can be implemented by a 3×3 convolution kernel, and the obtained dimensionality reduction result can have a channel dimension of 256.

[0115] The target center prediction module, target scale prediction module and target center offset prediction module are three parallel modules, which can be implemented by a 1×1 convolution kernel respectively.

[0116] When performing distributed training on the initial model, it is necessary to construct a true label for each module in the detector. For the true label of the target center point, since it is difficult to determine the exact center point position of an object, in order to reduce the uncertainty of a large number of negative samples around the positive sample, a two-dimensional Gaussian mask is applied to the location of each positive sample. , the formula is as follows:

[0117] (5)

[0118] (6)

[0119] in, Represents the center point position of the detector output, represents the number of targets in the sample image, The sample image is The center point coordinates, width and height of each target, and the variance of the two-dimensional Gaussian distribution and Proportional to the width and height of the target respectively.

[0120] The scale of an object can be defined as the width and height of the object, and for its true label, The position of the positive sample is assigned to target .

[0121] In order to reduce the error caused by point prediction, It is also assigned to all negative sample points within a radius of 2 from the positive sample point, and all other positions are assigned to 0.

[0122] The center point position of the sample image The position mapped to the output image , therefore, the target center point offset of the true label can be defined as:

[0123] (7)

[0124] in, is the downsampling factor, which is set to 4 in the embodiment of the present invention.

[0125] Furthermore, the loss function used by the detection model during training includes target center point loss , target scale loss and target center calibration offset loss ;

[0126] Target center point loss Calculated based on the Focal Loss loss function;

[0127] Target scale loss and target center calibration offset loss All are calculated based on the Smooth L1 loss function.

[0128] In the embodiment of the present invention, since the number of target center points in the sample image is less than that of non-target center points, the positive and negative samples are unbalanced, which is not conducive to the training of the initial model. Therefore, the Focal Loss function is used in the embodiment of the present invention to solve the problem of positive and negative sample imbalance. Calculated by the following formula:

[0129] (8)

[0130] (9)

[0131] (10)

[0132] in, represents the number of targets in the sample image, Represent the width and height of the sample image respectively, Represents the predicted coordinate point The probability of belonging to the target center point, are two hyperparameters, which are set to 、 , Indicates coordinate points Gaussian heatmap of .

[0133] use The loss function calculates the target scale loss and the target center point offset loss, and its formula is:

[0134] (11)

[0135] (12)

[0136] in, Represents the first The predicted and true values ​​of the target scale, Represents the first The predicted and true values ​​of the target center point offset.

[0137] In summary, the total loss function Defined as:

[0138] (13)

[0139] in, 、 、 They represent the weights of target center point loss, scale loss, and center point offset loss, respectively. In the embodiment of the present invention, they are set to 0.01, 1, and 0.1, respectively.

[0140] The detection model provided in the embodiment of the present invention can be defined as FGDF-Net. FGDF-Net is trained based on PyTorch and runs on four A100 PCIE-40GB-GPU devices. The backbone network (HRNet) of FGDF-Net uses the model weights pre-trained on the ImageNet dataset. In addition, for the target detection task mentioned in the embodiment of the present invention, the Adam optimization algorithm is adopted. On the pedestrian detection CityPersons dataset in target detection, the size of the sample image is set to 640×1280, the number of iterations in the training phase is set to 150, the batch size is set to 16, and the initial learning rate is set to multiply the learning rate by 0.1 every 50 iterations. The embodiment of the present invention uses the average missed detection rate ( ) as the evaluation indicator.

[0141] The CityPersons dataset is divided into three subsets: heavy occlusion subset (Heavy), partial occlusion subset (Parital), reasonable occlusion subset (Reasonable), and slight occlusion subset (Bare) based on pedestrian height and occlusion ratio. The division criteria of each subset are shown in Table 1.

[0142] Table 1. Division criteria of some subsets in the CityPersons dataset

[0143]

[0144] Table 2 shows the experimental results of the FGDF-Net provided in an embodiment of the present invention on the CityPersons dataset, and compares the average missed detection rate with existing object detection algorithms for occlusion. The average missed detection rate of FGDF-Net across all occlusion levels is lower than that of other algorithms. In particular, the average missed detection rate for the heavy occlusion subset is 43.4%, a 4.7% improvement compared to the CSP algorithm using the same backbone network (HRNet). These experimental results fully demonstrate the effectiveness of the fine-grained feature extraction and fusion module based on the attention mechanism in FGDF-Net for occluded object detection.

[0145] Table 2 Comparison of the average missed detection rate of each target detection algorithm in terms of occlusion

[0146]

[0147] like Figure 5 As shown, based on the above embodiment, an embodiment of the present invention provides an object detection device based on fine-grained feature extraction, including:

[0148] An image acquisition module 51 is used to acquire an image of a target to be detected;

[0149] The target detection module 52 is used to input the image to be detected into a detection model and obtain the detection result of the target output by the detection model;

[0150] The detection model includes a feature extraction module, a fine-grained feature extraction and fusion module, and a detector connected in sequence; the detection model is trained based on sample images carrying sample labels;

[0151] The feature extraction module is used to extract image features of multiple different resolutions of the image to be detected;

[0152] The fine-grained feature extraction and fusion module is configured to associate a first image feature with other image features among the multiple image features of different resolutions based on an attention mechanism, extract a target fine-grained feature based on the obtained association result, fuse the target fine-grained feature with the first image feature, and splice the obtained fusion result with the second image feature among the multiple image features of different resolutions in the channel dimension to obtain a splicing result;

[0153] The detector is used to obtain the detection result based on the splicing result.

[0154] On the basis of the above embodiments, in the object detection device based on fine-grained feature extraction provided in the embodiments of the present invention, the fine-grained feature extraction and fusion module is specifically used for:

[0155] Projecting the plurality of image features at different resolutions into a coding space, and calculating, in the coding space, an attention map for characterizing the correlation between the first image feature and the other image features;

[0156] Calculate the feature representations of all positions of the other image features, and perform weighted summation of the feature representations and the attention map to obtain the target fine-grained features.

[0157] On the basis of the above embodiment, in the target detection device based on fine-grained feature extraction provided in an embodiment of the present invention, the first image feature is the image feature with the lowest resolution among the multiple image features with different resolutions; and / or,

[0158] The second image feature is an image feature with the highest resolution among the multiple image features with different resolutions.

[0159] On the basis of the above embodiment, in the object detection device based on fine-grained feature extraction provided in the embodiment of the present invention, the number of the image features is 4;

[0160] The feature extraction module is HRNet which includes four feature extraction stages;

[0161] The image to be detected passes through the four feature extraction stages in sequence, and the fourth feature extraction stage outputs the image features.

[0162] On the basis of the above-mentioned embodiment, in the target detection device based on fine-grained feature extraction provided in the embodiment of the present invention, the detection model is obtained by performing distributed training on the initial model based on the sample image.

[0163] On the basis of the above embodiment, an object detection device based on fine-grained feature extraction is provided in an embodiment of the present invention, wherein the detector includes a dimensionality reduction module, a target center point prediction module, a target scale prediction module, and a target center point offset prediction module, wherein the dimensionality reduction module is connected to the target center point prediction module, the target scale prediction module, and the target center point offset prediction module, respectively;

[0164] The dimensionality reduction module is used to perform dimensionality reduction processing on the splicing result to obtain a dimensionality reduction result;

[0165] The target center point prediction module is used to determine the target center point based on the dimensionality reduction result; the target scale prediction module is used to determine the target scale based on the dimensionality reduction result; and the target center point offset prediction module is used to determine the target center point offset based on the dimensionality reduction result.

[0166] On the basis of the above embodiment, in the object detection device based on fine-grained feature extraction provided in the embodiment of the present invention, the detection result includes an object bounding box;

[0167] Accordingly, the detector is also used for:

[0168] Generate a target candidate frame based on the target center point and the target scale;

[0169] Based on the target center point offset, the target candidate box is fine-tuned to obtain the target bounding box.

[0170] Specifically, the functions of each module in the target detection device based on fine-grained feature extraction provided in the embodiment of the present invention correspond one-to-one to the operational procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.

[0171] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the target detection method based on fine-grained feature extraction provided in the above embodiments, the method including: obtaining an image to be detected of the target; inputting the image to be detected into the detection model to obtain the detection result of the target output by the detection model; wherein the detection model includes a feature extraction module, a fine-grained feature extraction and fusion module and a detector connected in sequence; the detection model is trained based on a sample image carrying a sample label; the feature extraction module is used to extract image features of multiple different resolutions of the image to be detected; the fine-grained feature extraction and fusion module is used to associate the first image feature of the multiple image features with other image features based on the attention mechanism, and extract the target fine-grained feature based on the obtained association result, fuse the target fine-grained feature with the first image feature, and splice the obtained fusion result with the second image feature of the multiple image features with different resolutions in the channel dimension to obtain a splicing result; the detector is used to obtain the detection result based on the splicing result.

[0172] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0173] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target detection method based on fine-grained feature extraction provided in the above embodiments, the method including: obtaining an image to be detected of the target; inputting the image to be detected into a detection model to obtain a detection result of the target output by the detection model; wherein the detection model includes a feature extraction module, a fine-grained feature extraction and fusion module and a detector connected in sequence; the detection model is trained based on a sample image carrying a sample label; the feature extraction module is used to extract image features of multiple different resolutions of the image to be detected; the fine-grained feature extraction and fusion module is used to associate the first image feature of the multiple image features with other image features based on an attention mechanism, and extract the target fine-grained feature based on the obtained association result, fuse the target fine-grained feature with the first image feature, and splice the obtained fusion result with the second image feature of the multiple image features with different resolutions in the channel dimension to obtain a splicing result; the detector is used to obtain the detection result based on the splicing result.

[0174] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the target detection method based on fine-grained feature extraction provided in the above-mentioned embodiments, the method comprising: obtaining an image to be detected of the target; inputting the image to be detected into a detection model to obtain a detection result of the target output by the detection model; wherein the detection model comprises a feature extraction module, a fine-grained feature extraction and fusion module and a detector connected in sequence; the detection model is trained based on a sample image carrying a sample label; the feature extraction module is used to extract image features of multiple different resolutions of the image to be detected; the fine-grained feature extraction and fusion module is used to associate a first image feature of the multiple image features with other image features of different resolutions based on an attention mechanism, and extract target fine-grained features based on the obtained association results, fuse the target fine-grained features with the first image features, and splice the obtained fusion results with the second image features of the multiple image features of different resolutions in the channel dimension to obtain a spliced ​​result; the detector is used to obtain the detection result based on the spliced ​​result.

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0176] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0177] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A target detection method based on fine-grained feature extraction, characterized in that: include: Obtain the image of the target to be detected; Inputting the image to be detected into a detection model to obtain a detection result of the target output by the detection model; The detection model includes a feature extraction module, a fine-grained feature extraction and fusion module, and a detector connected in sequence; the detection model is trained based on sample images carrying sample labels; The feature extraction module is used to extract image features of multiple different resolutions of the image to be detected; The fine-grained feature extraction and fusion module is configured to associate a first image feature with other image features among the multiple image features of different resolutions based on an attention mechanism, extract a target fine-grained feature based on the obtained association result, fuse the target fine-grained feature with the first image feature, and splice the obtained fusion result with the second image feature among the multiple image features of different resolutions in the channel dimension to obtain a splicing result; The detector is used to obtain the detection result based on the splicing result; The first image feature is an image feature with the lowest resolution among the multiple image features with different resolutions; and / or, The second image feature is an image feature with the highest resolution among the multiple image features with different resolutions.

2. The target detection method based on fine-grained feature extraction according to claim 1, characterized in that: The fine-grained feature extraction and fusion module is specifically used for: Projecting the plurality of image features at different resolutions into a coding space, and calculating, in the coding space, an attention map for characterizing the correlation between the first image feature and the other image features; Calculate the feature representations of all positions in the other image features, and perform weighted summation of the feature representations and the attention map to obtain the target fine-grained features.

3. The target detection method based on fine-grained feature extraction according to claim 1, characterized in that: The number of the image features is 4; The feature extraction module is HRNet which includes four feature extraction stages; The image to be detected passes through the four feature extraction stages in sequence, and the fourth feature extraction stage outputs the image features.

4. The target detection method based on fine-grained feature extraction according to claim 1, characterized in that: The detection model is obtained by performing distributed training on an initial model based on the sample image.

5. The target detection method based on fine-grained feature extraction according to any one of claims 1 to 4, characterized in that: The detector includes a dimensionality reduction module, a target center point prediction module, a target scale prediction module and a target center point offset prediction module, wherein the dimensionality reduction module is connected to the target center point prediction module, the target scale prediction module and the target center point offset prediction module respectively; The dimensionality reduction module is used to perform dimensionality reduction processing on the splicing result to obtain a dimensionality reduction result; The target center point prediction module is used to determine the target center point based on the dimensionality reduction result; The target scale prediction module is used to determine the target scale based on the dimensionality reduction result; The target center point offset prediction module is used to determine the target center point offset based on the dimensionality reduction result.

6. The target detection method based on fine-grained feature extraction according to claim 5, characterized in that: The detection result includes a target bounding box; Accordingly, the detector is also used for: Generate a target candidate frame based on the target center point and the target scale; Based on the target center point offset, the target candidate box is fine-tuned to obtain the target bounding box.

7. A target detection device based on fine-grained feature extraction, characterized in that: include: An image acquisition module is used to acquire an image of the target to be detected; The target detection module is used to input the image to be detected into the detection model and obtain the detection result of the target output by the detection model; The detection model includes a feature extraction module, a fine-grained feature extraction and fusion module, and a detector connected in sequence; the detection model is trained based on sample images carrying sample labels; The feature extraction module is used to extract image features of multiple different resolutions of the image to be detected; The fine-grained feature extraction and fusion module is configured to associate a first image feature with other image features among the multiple image features of different resolutions based on an attention mechanism, extract a target fine-grained feature based on the obtained association result, fuse the target fine-grained feature with the first image feature, and splice the obtained fusion result with the second image feature among the multiple image features of different resolutions in the channel dimension to obtain a splicing result; The detector is used to obtain the detection result based on the splicing result; The first image feature is an image feature with the lowest resolution among the multiple image features with different resolutions; and / or, The second image feature is an image feature with the highest resolution among the multiple image features with different resolutions.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the target detection method based on fine-grained feature extraction according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the object detection method based on fine-grained feature extraction according to any one of claims 1 to 6 is implemented.

Images