Cross-modal target detection method and system based on multi-scale features

Abstract

The present invention discloses a cross-modal target detection method and system based on multi-scale features, including: obtaining an image to be detected and a text description of the target to be detected; inputting the image to be detected and the text description of the target to be detected into a trained cross-modal target detection model; outputting the target detection result; wherein the cross-modal target detection model performs target detection by fusing text features with visual features at multiple scales to improve the perception of target entities. For the "cross-modal target detection" task, target detection is performed by fusing text features with visual features at multiple scales, which improves the algorithm's perception of target entities, thereby further improving the effect of the model in the "cross-modal target retrieval" task.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a cross-modal target detection method and system based on multi-scale features. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] The general object detection technology is to find all objects of predefined categories in a given image and give the coordinates of the corresponding objects. With the gradual promotion and application of deep learning technology, in some scenarios, the algorithm needs to be able to perceive unknown categories or locate the position of the object based on vague attribute descriptions (for example, finding a man in a brown shirt in a surveillance video). The current cross-modal object detection task is mainly divided into single-stage and two-stage algorithms. For the two-stage algorithm, given an input image, the first step is to generate candidate regions using a target detection network pre-trained on a large dataset. The second step is to rank the candidate regions according to the language query about the image. The performance of the two-stage method depends largely on the results of the first stage. If the candidate region in the first stage does not contain the object, it will be difficult to capture the corresponding object in the second stage. In addition, the region proposal module brings huge computational costs, so these methods are difficult to use in actual systems. The single-stage-based method directly predicts the bounding box of the target by fusing image-text features. Due to the simplicity of the framework and the absence of a region proposal module, the single-stage method can achieve faster reasoning speed and higher accuracy.

[0004] Existing cross-modal object detection algorithms are insufficient in their ability to perceive long natural sentence queries and multi-scale objects, which limits the further improvement of model performance. The natural language encoding module of most methods encodes text into a single natural language vector, which easily leads to the loss of some language details when dealing with long natural language encoding tasks. Most methods tend to perform object detection tasks at a single scale, which can easily cause mismatching of objects and text when the object has a large deformation. Summary of the invention

[0005] In order to address the shortcomings of the prior art, the present invention provides a cross-modal target detection method and system based on multi-scale features; for the "cross-modal target detection" task, target detection is performed by fusing text features with visual features at multiple scales, thereby improving the algorithm's perception ability of target entities, thereby further improving the effect of the model in the "cross-modal target retrieval" task.

[0006] In a first aspect, the present invention provides a cross-modal target detection method based on multi-scale features;

[0007] Cross-modal target detection methods based on multi-scale features include:

[0008] Obtain an image of a target to be detected and a text description of the target to be detected;

[0009] The image to be detected and the text description of the target to be detected are input into the trained cross-modal target detection model; and the target detection result is output; wherein the cross-modal target detection model performs target detection by fusing text features with visual features at multiple scales to improve the perception ability of the target entity.

[0010] In a second aspect, the present invention provides a cross-modal target detection system based on multi-scale features;

[0011] A cross-modal target detection system based on multi-scale features, including:

[0012] An acquisition module is configured to: acquire an image of a target to be detected and a text description of the target to be detected;

[0013] A detection module is configured to: input an image to be detected and a text description of the target to be detected into a trained cross-modal target detection model; and output a target detection result; wherein the cross-modal target detection model performs target detection by fusing text features with visual features at multiple scales to improve the perception of the target entity.

[0014] In a third aspect, the present invention further provides an electronic device, comprising:

[0015] a memory for non-transitory storage of computer-readable instructions; and

[0016] a processor for executing the computer readable instructions,

[0017] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0018] In a fourth aspect, the present invention further provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0019] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] This proposal is based on a trained cross-modal target detection model. On the basis of fine-grained modeling of natural language, it performs detection tasks on multiple image-text features, effectively improving the ability to perceive the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 This is a flow chart of a cross-modal target detection method based on multi-scale features of Example 1. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0025] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0027] In this embodiment, all data is obtained in compliance with laws and regulations and based on the user's consent, and is used legally.

[0028] Embodiment 1

[0029] This embodiment provides a cross-modal target detection method based on multi-scale features;

[0030] like Figure 1 As shown in FIG. 1 , a cross-modal target detection method based on multi-scale features includes:

[0031] S101: Acquire an image of a target to be detected and a text description of the target to be detected;

[0032] S102: Inputting the image to be detected and the text description of the target to be detected into the trained cross-modal target detection model; outputting the target detection result; wherein the cross-modal target detection model performs target detection by fusing text features with visual features at multiple scales to improve the perception ability of the target entity.

[0033] Furthermore, the cross-modal object detection model includes:

[0034] A feature encoding module is used to extract features from an input image to be detected and extract feature maps of the first, second and third scales;

[0035] A feature flattening module, used to flatten the feature maps of the first, second and third scales respectively;

[0036] A spatial position adding module is used to add spatial position information to the flattened first, second and third scale feature maps to obtain first, second and third scale features with added spatial position information respectively;

[0037] A splicing module, used to sequentially splice the first scale feature with added spatial position information and the third scale feature with added spatial position information to obtain a feature after the first scale feature and the third scale feature are spliced;

[0038] Transformer Decoder module, used to use the second-scale feature with added spatial position information as the query of Transformer Decoder, and the concatenated features of the first-scale feature and the third-scale feature as the key and value of Transformer Decoder. Transformer Decoder outputs a visual feature sequence that integrates multi-scale information;

[0039] A feature fusion module is used to combine text features with a visual feature sequence that integrates multi-scale information to obtain a visual text feature sequence;

[0040] The object detection module is used to obtain the object detection box based on the visual text feature sequence.

[0041] Furthermore, the feature encoding module uses Dartnet-53 as the feature extraction network.

[0042] Furthermore, the first scale is 1 / 8 of the size of the image to be detected, the second scale is 1 / 16 of the size of the image to be detected, and the third scale is 1 / 32 of the size of the image to be detected.

[0043] Assuming the image size is 224*224, then 1 / 8 represents a size of 28*28, 1 / 16 represents a size of 14*14, and 1 / 32 represents a size of 7*7.

[0044] Exemplarily, the feature flattening module is used to flatten the feature maps of the first, second and third scales; Flatten the feature maps of the first, second and third scales in the spatial dimension to obtain three one-dimensional feature sequences.

[0045] Among them, the size of the first scale feature map is 28*28*C2. After the Flatten operation, the feature map becomes a one-dimensional sequence of 784*C2; the size of the second scale feature map is 14*14*C1. After the Flatten operation, the feature map becomes a one-dimensional sequence of 196*C1; the size of the third scale feature map is 7*7*C3. After the Flatten operation, the feature map becomes a one-dimensional sequence of 49*C3. Then, the features of these three scales are unified into the feature number C2 of the intermediate scale through the linear layer, and three feature sequences of 196*C2, 784*C2, and 49*C2 are obtained.

[0046] Furthermore, the spatial position information is set as a randomly initialized parameter and is obtained by adaptive learning of the Transformer network.

[0047] Furthermore, the target detection module is used to obtain a target detection frame according to the visual text feature sequence; the specific working process includes:

[0048] Adjust the visual text feature sequence into a 2D feature map;

[0049] In each square of the 2D feature map, 9 anchor boxes are assigned;

[0050] Based on the anchor frame, regress the location of the target described by natural language and the confidence of whether the target is included;

[0051] The predicted box with the highest confidence is taken as the target detection box.

[0052] Furthermore, the text features are obtained by extracting features from the input text using a pre-trained BERT model.

[0053] Furthermore, the BERT model is established using the Transformer encoder.

[0054] Furthermore, the trained cross-modal object detection model; the specific training process includes:

[0055] Constructing a training set; wherein the training set is an image with a known target detection result label;

[0056] The training set is input into the cross-modal target detection model, and the cross-modal target detection model is trained. When the loss function value no longer decreases, the training is stopped to obtain the trained cross-modal target detection model.

[0057] Embodiment 2

[0058] This embodiment provides a cross-modal target detection system based on multi-scale features;

[0059] A cross-modal target detection system based on multi-scale features, including:

[0060] An acquisition module is configured to: acquire an image of a target to be detected and a text description of the target to be detected;

[0061] A detection module is configured to: input an image to be detected and a text description of the target to be detected into a trained cross-modal target detection model; and output a target detection result; wherein the cross-modal target detection model performs target detection by fusing text features with visual features at multiple scales to improve the perception of the target entity.

[0062] It should be noted that the acquisition module and the detection module correspond to steps S101 to S102 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0063] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0064] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0065] Embodiment 3

[0066] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0067] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0068] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0069] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0070] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0071] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0072] Embodiment 4

[0073] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cross-modal target detection method based on multi-scale features, characterized by: include: Obtain an image of a target to be detected and a text description of the target to be detected; Input the image to be detected and the text description of the target to be detected into the trained cross-modal target detection model; output the target detection result; wherein the cross-modal target detection model performs target detection by fusing text features with visual features at multiple scales to improve the perception ability of the target entity; the cross-modal target detection model includes: A feature encoding module is used to extract features from an input image to be detected and extract feature maps of the first, second and third scales; The feature flattening module is used to flatten the feature maps of the first, second and third scales; the feature maps of the first, second and third scales are flattened in the spatial dimension to obtain three one-dimensional feature sequences; A spatial position adding module is used to add spatial position information to the flattened first, second and third scale feature maps to obtain first, second and third scale features with added spatial position information respectively; A splicing module, used to perform feature splicing on the first scale feature with added spatial position information and the third scale feature with added spatial position information, to obtain a feature after the first scale feature and the third scale feature are spliced; The Transformer Decoder module is used to use the second-scale feature with added spatial position information as the query of the Transformer Decoder, and the concatenated features of the first-scale feature and the third-scale feature as the key and value of the Transformer Decoder. The Transformer Decoder outputs a visual feature sequence that integrates multi-scale information; A feature fusion module is used to combine text features with a visual feature sequence that integrates multi-scale information to obtain a visual text feature sequence; The object detection module is used to obtain the object detection box based on the visual text feature sequence.

2. The cross-modal target detection method based on multi-scale features according to claim 1, characterized in that: The feature encoding module uses Dartnet-53 as the feature extraction network.

3. The cross-modal target detection method based on multi-scale features as claimed in claim 1, characterized in that: The spatial position information is set as randomly initialized parameters and is obtained by adaptive learning of the Transformer network.

4. The cross-modal target detection method based on multi-scale features as claimed in claim 1, characterized in that: The target detection module is used to obtain a target detection frame according to a visual text feature sequence; the specific working process includes: The visual text feature sequence is adjusted to a 2D feature map; 9 anchor boxes are assigned to each square of the 2D feature map; based on the anchor boxes, the position of the target described by the natural language and the confidence of whether the target is included are regressed; the predicted box with the highest confidence is used as the target detection box.

5. The cross-modal target detection method based on multi-scale features as claimed in claim 1, characterized in that: The text features are obtained by extracting features from input text using a pre-trained BERT model.

6. A cross-modal target detection system based on multi-scale features using the cross-modal target detection method based on multi-scale features as claimed in claim 1, characterized in that: include: An acquisition module is configured to: acquire an image of a target to be detected and a text description of the target to be detected; A detection module is configured to: input an image to be detected and a text description of the target to be detected into a trained cross-modal target detection model; and output a target detection result; wherein the cross-modal target detection model performs target detection by fusing text features with visual features at multiple scales to improve the perception of the target entity.

7. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 5 is executed.

8. A storage medium, characterized in that: The computer-readable instructions are non-transitorily stored, wherein when the computer-readable instructions are executed by a computer, the method according to any one of claims 1 to 5 is performed.

Images