Semantic data driven weak and small target visual detection method and detection equipment

Abstract

The invention discloses a semantic data-driven weak and small target visual detection method and detection equipment, multi-modal features of a target image are identified based on a detection model to obtain bounding box coordinates of a weak and small target, and the detection model comprises a visual network, a semantic network, a multi-stage feature interaction enhancement module and a cross-modal information fusion decoder. The semantic network extracts semantic features based on semantic information of an image, the visual network extracts visual features of the image based on the semantic features, and the multi-level feature interaction enhancement module processes the semantic features and the visual features to obtain high-order semantic features and high-order visual features; the cross-modal information fusion decoder fuses the high-order semantic features and the high-order visual features to obtain multi-modal features of the target image, semantic information is utilized to perform information complementation on the visual features, multi-modal information fusion in a weak and small target visual detection scene is realized, and weak and small target detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a semantic data-driven small target visual detection method and detection device. Background Art

[0002] Small targets, such as drones, are often difficult to effectively identify using visual detection technology because of their low flying altitude, slow speed, and small size. These targets occupy a small proportion of pixels in the image, have low signal strength, and are easily submerged by the background image. Existing small target detection methods face some challenges: First, pure visual detection methods rely on a single data source and cannot make full use of other auxiliary information to improve detection accuracy and robustness. Second, existing visual feature extraction networks often cannot adapt to the needs of multi-source heterogeneous data fusion. Summary of the invention

[0003] The present invention provides a semantic data driven small target visual detection method and detection device to solve the problems raised in the above background technology.

[0004] In the first aspect, a semantic data-driven small object visual detection method is provided, comprising:

[0005] The detection model is based on identifying the multimodal features of the target image to obtain the bounding box coordinates of the weak target. The detection model includes a visual network, a semantic network, a multi-level feature interaction enhancement module and a cross-modal information fusion decoder. The semantic network extracts semantic features based on the semantic information of the image, and the visual network extracts visual features of the image based on the semantic features. The multi-level feature interaction enhancement module processes the semantic features and the visual features to obtain high-order semantic features and high-order visual features. The cross-modal information fusion decoder fuses the high-order semantic features and the high-order visual features to obtain the multimodal features of the target image.

[0006] Furthermore, the semantic network provides adaptive weights and biases for the visual network through a text interaction attention module.

[0007] Furthermore, the feature map of the i-th stage of the middle layer of the pre-trained visual feature extraction network is extracted based on the text interaction attention module. Modulation is performed to dynamically generate adaptive weights and biases related to semantic features.

[0008] Furthermore, dynamically generating weights and biases related to the semantic features includes: performing average pooling aggregation on the semantic feature information, and obtaining weights and biases through a tanh activation function on the semantic feature information after average pooling aggregation.

[0009] Furthermore, processing the semantic features and visual features to obtain high-order semantic features and high-order visual features includes: linearly projecting the visual features and semantic features to the same dimension through a fully connected layer, adding position embedding to obtain first-order visual features and first-order semantic features, and performing multi-level fusion of the first-order visual features and first-order semantic features through a self-attention module and a cross-attention module to obtain high-order visual features and high-order semantic features.

[0010] Furthermore, the cross-modal information fusion decoder fuses high-order semantic features and high-order visual features through a cross-attention module to obtain the multimodal features of the target image.

[0011] Furthermore, the semantic feature is represented as F l ∈R l×d , where l and d represent the token number and feature dimension of the semantic feature.

[0012] Furthermore, the visual feature is represented as F v ∈R h×w×c , where h, w, and c represent the height, width, and number of channels of the visual feature, respectively.

[0013] In a second aspect, a semantic data-driven small target visual detection device is provided, comprising a processor and a memory, wherein the memory is used to store computer instructions. When the processor executes the instructions, the detection device executes the method described in the first aspect.

[0014] According to a third aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the method described in the first aspect.

[0015] The present invention incorporates semantic data into the visual detection method for small and weak targets, completes the extraction and fusion of multi-source heterogeneous data features, uses the abstract and condensed physical meaning behind the semantic information to complement the visual features, and realizes multimodal information fusion in the small and weak target visual detection scenario, thereby enhancing the visual detection efficiency of small and weak targets and improving the detection accuracy of small and weak targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is the overall composition diagram of the detection model;

[0017] Figure 2 A block diagram of the text interaction attention module of the detection model;

[0018] Figure 3 Block diagram of the multi-level feature interaction fusion decoder composition for the detection model. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0020] In a first aspect, an embodiment of the present application provides a semantic data-driven small target visual detection method, including:

[0021] Based on the detection model, the multimodal features of the target image are identified to obtain the bounding box coordinates of the weak target, such as Figure 1 As shown, the detection model includes a visual network, a semantic network, a multi-level feature interaction enhancement module and a cross-modal information fusion decoder. The semantic network extracts semantic features based on the semantic information of the image, and the visual network extracts visual features of the image based on the semantic features. The multi-level feature interaction enhancement module processes the semantic features and visual features to obtain high-order semantic features and high-order visual features. The cross-modal information fusion decoder fuses the high-order semantic features and high-order visual features to obtain multi-modal features of the target image. The semantic network provides adaptive weights and biases for the visual network through the text interaction attention module, that is, the feature map of the i-th stage of the intermediate layer of the pre-trained visual feature extraction network is obtained. Modulation is performed to dynamically generate adaptive weights and biases related to semantic features.

[0022] Specifically, given the semantic information of an image, the extracted semantic features are l ∈R l×d , where l and d represent the token number and feature dimension of the semantic feature; then, the semantic feature F l Feed it to the text interaction attention module to dynamically generate weights and biases for the visual extraction network; secondly, given a visual image I, extract visual features F v ∈R h×w×c , where h, w, c represent the height, width and number of channels of the visual feature respectively; then, the visual feature F enhanced by the semantic feature v and semantic features F l The cross-modal information fusion decoder interactively fuses the semantic features and enhanced visual features by linear projection mapping to the same dimension. Finally, the bounding box coordinates of the weak target are inferred by the localization module.

[0023] The visual network, semantic network, multi-level feature interaction enhancement module and cross-modal information fusion decoder are introduced in detail below.

[0024] 1. Visual Web and Semantic Web

[0025] The visual network mainly realizes the feature extraction of visual image information. In the middle layer of the pre-trained visual feature extraction network, a text interaction attention module is added to improve the visual feature extraction efficiency under the guidance of semantic features. The core of this small target visual detection method is to add semantic information to the visual network to dynamically adjust the visual feature extraction process, which is conducive to the alignment of visual features and semantic features to accurately locate small targets.

[0026] Pure image visual localization methods usually use a pre-trained visual feature extraction network to directly encode the input image into visual features. However, due to the lack of prior knowledge of the object referred to in the text description, the extracted visual features contain the features of all similar objects in the image, resulting in low accuracy of the localization results.

[0027] Conditional Batch Normalization (CBN) uses text information to learn the two feature affine transformation parameters (i.e., γ c and β c ), used to modulate the visual feature F v , as shown in formula (1).

[0028] CBN(F i |γ c ,β c )=γ c F i +β c (1)

[0029] However, CBN lacks sufficient flexibility and is only used between convolutional layers and activation functions. At the same time, CBN only focuses on the channel dimension of visual features and lacks attention to the spatial information of the described target, while the precise positioning of weak targets is highly dependent on spatial information.

[0030] like Figure 2 As shown, in order to guide the visual feature extraction network, the text interaction attention module maps the feature map of the i-th stage of the visual feature extraction network Modulate and dynamically generate adaptive weights and biases related to semantic features, namely channel weights Space rights and bias B i .

[0031] For channel weights and bias B i , aggregates information from semantic features through average pooling, and uses a fully connected layer to map this information to the same dimension as the visual feature channel. Then, the tanh activation function is applied to obtain the channel weights and bias B i∈[-1,1], and use the fully connected layer to further refine the features, as shown in formula (2).

[0032]

[0033] Spatial Weight A similar process as in formula (2) is used to learn visual features related to the text description. Subsequently, a deconvolution (DeConv) layer is used to dynamically generate visual features. The advantage of the deconvolution layer is that it can learn parameters to dynamically adapt to the feature extraction process, retain more spatial dimension information, and have fewer parameters, as shown in formula (3).

[0034]

[0035] In the formula, the spatial weight The spatial weights generated based on the semantic features Channel weight and bias B i Replace γ in formula (1) c and β c , dynamically adjust the visual feature extraction process, as shown in formula (4).

[0036]

[0037] The size of the weights and biases determines the importance of each channel and spatial dimension in the visual feature. Therefore, the text interaction attention module can selectively highlight or suppress visual features based on semantic information, directing the attention of the visual feature extraction network to the described target. In addition, since the text interaction attention module has only three parameters per feature map (i.e. and B i ), which is a scalable and computationally efficient modulation method.

[0038] 2. Multi-level feature interaction fusion decoder

[0039] The multi-level feature interaction fusion decoder includes a multi-level feature interaction enhancement module and a cross-modal information fusion decoder. The purpose of the multi-level feature interaction fusion decoder is to reduce the ambiguity of the target description in the reasoning process and to aggregate and enhance the visual features of weak targets using multi-layer cross-modal interactions.

[0040] Given a visual feature F v ∈R h×w×c and semantic features F l ∈R l×d, use the fully connected layer to perform linear projection to the same dimension, and add position embedding. Position embedding uses the cos encoding in Transformer, that is, on the basis of the original data vector, add a code that is strongly related to the position information so that the original feature vector contains the position information. The formula is as follows:

[0041]

[0042] Where pos(·) is the position embedding function. v ',F l '∈R a×b , where a and b represent the number of symbols and the dimension. The visual features F are fused through self-attention and cross-attention modules v and semantic features F l Obtain the visual semantic feature F after semantic feature enhancement + .

[0043]

[0044] In the formula, MHSA(·) and MHCA(·) are self-attention and cross-attention modules respectively. Using the cross-attention module, F l ' and F + Combined to obtain the characteristic attributes of the described target, the formula is as follows.

[0045] F m =MHCA(F + ,F l ') (7)

[0046] Among them, the multimodal feature F m Finally, it is used for target positioning. The cross-modal interaction of semantic features and visual features helps to obtain richer feature information of weak targets, reduce ambiguity in visual positioning, and establish stronger visual semantic connections. The structure of the multi-level feature interaction fusion decoder is as follows: Figure 3 shown.

[0047] In the second aspect, a detection device is provided in an embodiment of the present application, including: a processor and a memory. The number of processors can be one or more, and the memory is used to store computer instructions. When the processor executes the instructions, the detection device performs the steps of the small target visual detection method.

[0048] In a third aspect, a computer program product is provided in an embodiment of the present application, including a computer program / instruction, which implements the steps of a method for visual detection of small and dim targets when executed by a processor.

[0049] Those of ordinary skill in the art will appreciate that the various method steps and units described in the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0050] The methods or steps described in conjunction with the embodiments disclosed herein may be implemented by hardware, a software program executed by a processor, or a combination of the two. The software program may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0051] Although the present invention is described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions should all be within the scope of the present invention.

Claims

1. A semantic data driven small and dim object visual detection method, characterized in that: The multimodal features of the target image are identified based on a detection model to obtain the bounding box coordinates of the weak target. The detection model includes a visual network, a semantic network, a multi-level feature interaction enhancement module and a cross-modal information fusion decoder. The semantic network extracts semantic features based on the semantic information of the image, the visual network extracts visual features of the image based on the semantic features, the multi-level feature interaction enhancement module processes the semantic features and the visual features to obtain high-order semantic features and high-order visual features, and the cross-modal information fusion decoder fuses the high-order semantic features and the high-order visual features to obtain the multimodal features of the target image.

2. The method according to claim 1, characterized in that The visual network extracts visual features of an image based on the semantic features, including: the semantic network provides adaptive weights and biases for the visual network through a text interaction attention module.

3. The method according to claim 2, characterized in that The semantic network provides adaptive weights and biases for the visual network through a text interaction attention module, including: modulating the feature map of the i-th stage of the middle layer of the pre-trained visual feature extraction network based on the text interaction attention module, and dynamically generating adaptive weights and biases related to semantic features.

4. The method according to claim 3, characterized in that The dynamic generation of adaptive weights and biases related to semantic features includes: performing average pooling aggregation on the semantic feature information, and obtaining adaptive weights and biases through a tanh activation function on the semantic feature information after average pooling aggregation.

5. The method according to claim 1, characterized in that The processing of semantic features and visual features to obtain high-order semantic features and high-order visual features includes: linearly projecting the visual features and semantic features to the same dimension through a fully connected layer, adding position embedding to obtain first-order visual features and first-order semantic features, and performing multi-level fusion of the first-order visual features and first-order semantic features through a self-attention module and a cross-attention module to obtain high-order visual features and high-order semantic features.

6. The method according to claim 1, characterized in that The cross-modal information fusion decoder fuses high-order semantic features and high-order visual features through a cross-attention module to obtain multimodal features of the target image.

7. The method according to claim 1, characterized in that The semantic features are expressed as Among them, and d represent the token number and feature dimension of the semantic feature.

8. The method according to claim 1, characterized in that The visual features are expressed as Where h, w, and c represent the height, width, and number of channels of the visual feature, respectively.

9. A semantic data driven small target visual detection device, characterized in that: include: Memory: used to store a computer program for implementing the detection method according to any one of claims 1 to 8; Processor: used to implement the detection method described in any one of claims 1 to 8 when executing the computer program.

10. A computer program product comprising computer program instructions, the computer program instructions enabling a computer to perform the method according to any one of claims 1 to 8.

Images