A cross-modal image object detection method and device using text information

Abstract

The application relates to a cross-modal image target detection method and device using text information, and the method comprises the following steps: acquiring image data and label information of a target to be recognized; adopting a candidate frame neural network module to detect all targets in the image data, and performing interception to obtain a plurality of regions of interest; adopting an image feature extraction neural network module to extract image features of the regions of interest, and obtaining first features; adopting a text feature extraction neural network module to extract text features of the label information of the target to be recognized, and obtaining second features; merging the first features and the second features to obtain a merged matrix; adopting a cross-modal feature fusion neural network module to interactively fuse the features in the merged matrix, and separating the fused matrix into two separated matrices; calculating a similarity matrix of the two separated matrices, and determining the category of target detection according to the similarity matrix. The application makes up for the defect of the detection capability of an open vocabulary.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a cross-modal image target detection method and apparatus utilizing text information. Background Technology

[0002] General object detection methods have two limitations: 1. They only utilize image information and cannot effectively use text information to increase the semantic information of the image; 2. They cannot arbitrarily expand the categories to be detected. For example, if the model only has 10 categories during training, then the detected targets will only be included in 10 categories during inference. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a cross-modal image target detection method and apparatus that utilizes text information, thereby overcoming the deficiencies in detection capabilities based on open lexicons.

[0004] The technical solution adopted by this invention to solve its technical problem is: to provide a cross-modal image target detection method utilizing text information, comprising the following steps:

[0005] Acquire image data and label information of the target to be identified;

[0006] A candidate box neural network module is used to detect all targets in the image data and determine the position of each target. Based on the position, each target is extracted from the image data to obtain multiple regions of interest.

[0007] An image feature extraction neural network module is used to extract image features from the region of interest to obtain the first feature;

[0008] A text feature extraction neural network module is used to extract text features from the label information of the target to be identified, thereby obtaining a second feature;

[0009] The first feature and the second feature are combined to obtain a merged matrix;

[0010] A cross-modal feature fusion neural network module is used to interactively fuse the first and second features in the merged matrix to obtain a fusion matrix, and then the fusion matrix is ​​separated into two separation matrices.

[0011] Calculate the similarity matrix between the two separation matrices, and determine the category of the detected object based on the similarity matrix.

[0012] The candidate box neural network module is a DETR object detection network.

[0013] The image feature extraction neural network module is a VIT pre-trained model.

[0014] The text feature extraction neural network module is a BERT network.

[0015] The cross-modal feature fusion neural network module includes three self-attention network layers and two fully connected layers arranged sequentially. The three self-attention network layers are used to interactively fuse the first feature and the second feature in the fusion matrix and output a fusion matrix. The two fully connected layers are used to separate the fusion matrix into two separation matrices.

[0016] The technical solution adopted by this invention to solve its technical problem is: to provide a cross-modal image target detection device utilizing text information, comprising:

[0017] The acquisition module is used to acquire image data and label information of the target to be identified;

[0018] The candidate box neural network module is used to detect all targets in the image data, determine the position of each target, and extract each target from the image data based on the position to obtain multiple regions of interest;

[0019] An image feature extraction neural network module is used to extract image features from the region of interest to obtain a first feature;

[0020] The text feature extraction neural network module is used to extract text features from the label information of the target to be identified, and obtain the second feature;

[0021] The merging module is used to merge the first feature and the second feature to obtain a merge matrix;

[0022] A cross-modal feature fusion neural network module is used to interactively fuse the first feature and the second feature in the merge matrix to obtain a fusion matrix, and then separate the fusion matrix into two separation matrices;

[0023] The classification module is used to calculate the similarity matrix between the two separation matrices and determine the category of the detected object based on the similarity matrix.

[0024] The cross-modal feature fusion neural network module includes three self-attention network layers and two fully connected layers arranged sequentially. The three self-attention network layers are used to interactively fuse the first feature and the second feature in the fusion matrix and output a fusion matrix. The two fully connected layers are used to separate the fusion matrix into two separation matrices.

[0025] The classification module includes: a calculation unit, used to perform matrix multiplication on two separation matrices and perform Soft-Max normalization on the result to obtain a similarity matrix; and a comparison unit, used to compare the similarity of the labels in the similarity matrix with a threshold, remove labels with similarity less than the threshold, and use the retained labels as the category of the object detection.

[0026] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned cross-modal image target detection method using text information.

[0027] The technical solution adopted by the present invention to solve its technical problem is: to provide a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above-mentioned cross-modal image target detection method using text information are implemented.

[0028] Beneficial effects

[0029] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: The present invention utilizes the combined effect of the candidate box neural network module, the image feature extraction neural network module, the text feature extraction neural network module and the cross-modal feature fusion neural network module to increase the semantic information of image features, and can effectively achieve the detection capability of any category without increasing category training, thus making up for the deficiency of detection capability in open lexicon. Attached Figure Description

[0030] Figure 1 This is a flowchart of the first embodiment of the present invention;

[0031] Figure 2 This is a structural block diagram of the second embodiment of the present invention. Detailed Implementation

[0032] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0033] The first embodiment of the present invention relates to a cross-modal image target detection method utilizing textual information, such as... Figure 1 As shown, it includes the following steps:

[0034] Step 1: Obtain image data and label information of the target to be identified;

[0035] Step 2: Use a candidate box neural network module to detect all targets in the image data, determine the position of each target, and extract each target from the image data based on the position to obtain multiple regions of interest.

[0036] In this step, the candidate box neural network module uses the DETR object detection network, which can locate M targets in the image data and output M coordinate information. Each coordinate information is the x-axis top-left corner coordinate, y-axis top-left corner coordinate, x-axis bottom-right corner coordinate, and y-axis bottom-right corner coordinate of the bounding box containing the target. By using the M coordinate information, the target region is extracted from the original input image, thus obtaining the region of interest of the target.

[0037] Step 3: Use the image feature extraction neural network module to extract image features from the region of interest to obtain the first feature.

[0038] In this step, the image feature extraction neural network module uses the VIT pre-trained model, which can extract features of the ROI, thereby obtaining M first features. The M first features constitute the first feature vector, and the length of the first feature vector is 768dim, named A.

[0039] Step 4: Use the text feature extraction neural network module to extract text features from the label information of the target to be identified, and obtain the second feature.

[0040] In this step, the text feature extraction neural network module uses the BERT network. Since the text feature extraction neural network module is used to extract text features, when the label information of the target to be identified is used as its input, the label information of the target to be identified needs to be merged into a sentence. For example, if the label information of the target to be identified includes categories such as bird, duck, and car, these categories are used as the result of sentence segmentation and input into the text feature extraction neural network module. Through this text feature extraction neural network module, the features of the input text can be extracted. N second features are formed by N second features, and the length of the second feature vector is 768dim, named B.

[0041] Step 5: Merge the first feature and the second feature to obtain a merged matrix, that is, A+B, to obtain a merged matrix with shape (M+N, 768);

[0042] Step 6: Use the cross-modal feature fusion neural network module to interactively fuse the first feature and the second feature in the merged matrix to obtain a fusion matrix, and then separate the fusion matrix into two separate matrices.

[0043] In this step, the cross-modal feature fusion neural network module includes three self-attention network layers and two fully connected layers arranged sequentially. The three self-attention network layers are used to interactively fuse the first and second features in the merging matrix, outputting a fusion matrix, that is, through the characteristics of the self-attention network layers, interactive image features and text features, the output is a (M+N, 768) fusion matrix, named C; the two fully connected layers are used to separate the fusion matrix into two separation matrices, that is, the fusion matrix C is used as the input of the fully connected layer to obtain a (M+N, 512) matrix, which is then separated to obtain a (M, 512) and a (N, 512) separation matrix, named D and E respectively.

[0044] Step 7: Calculate the similarity matrix between the two separation matrices and determine the target detection category based on the similarity matrix. Specifically, perform matrix multiplication on separation matrix D and separation matrix E, followed by Soft-Max normalization to obtain a similarity matrix of shape (M, N), representing the similarity between M images and N labels. Assuming a threshold of 0.5, filter out category labels corresponding to similarity scores less than 0.5, retain category labels with similarity scores greater than or equal to 0.5, and use the retained labels as the final target detection category.

[0045] It is easy to see that the present invention utilizes the combined effects of a candidate box neural network module, an image feature extraction neural network module, a text feature extraction neural network module, and a cross-modal feature fusion neural network module to increase the semantic information of image features. It can effectively achieve the detection capability of any category without increasing the training of categories, thus making up for the deficiency of detection capability in open lexicons.

[0046] The second embodiment of the present invention relates to a cross-modal image target detection device utilizing textual information, such as... Figure 2 As shown, it includes:

[0047] The acquisition module is used to acquire image data and label information of the target to be identified;

[0048] The candidate box neural network module is used to detect all targets in the image data, determine the position of each target, and extract each target from the image data based on the position to obtain multiple regions of interest;

[0049] An image feature extraction neural network module is used to extract image features from the region of interest to obtain a first feature;

[0050] The text feature extraction neural network module is used to extract text features from the label information of the target to be identified, and obtain the second feature;

[0051] The merging module is used to merge the first feature and the second feature to obtain a merge matrix;

[0052] A cross-modal feature fusion neural network module is used to interactively fuse the first feature and the second feature in the merge matrix to obtain a fusion matrix, and then separate the fusion matrix into two separation matrices;

[0053] The classification module is used to calculate the similarity matrix between the two separation matrices and determine the category of the detected object based on the similarity matrix.

[0054] The candidate box neural network module is a DETR object detection network.

[0055] The image feature extraction neural network module is a VIT pre-trained model.

[0056] The text feature extraction neural network module is a BERT network.

[0057] The cross-modal feature fusion neural network module includes three self-attention network layers and two fully connected layers arranged sequentially. The three self-attention network layers are used to interactively fuse the first feature and the second feature in the fusion matrix and output a fusion matrix. The two fully connected layers are used to separate the fusion matrix into two separation matrices.

[0058] The classification module includes: a calculation unit, used to perform matrix multiplication on two separation matrices and perform Soft-Max normalization on the result to obtain a similarity matrix; and a comparison unit, used to compare the similarity of the labels in the similarity matrix with a threshold, remove labels with similarity less than the threshold, and use the retained labels as the category of the object detection.

[0059] A third embodiment of the present invention relates to an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the cross-modal image target detection method utilizing text information of the first embodiment.

[0060] The fourth embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the cross-modal image target detection method utilizing text information of the first embodiment.

[0061] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0062] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0066] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A cross-modal image object detection method using text information, characterized by, The method comprises the following steps: obtaining image data and label information of a target to be recognized; detecting all targets in the image data by using a candidate box neural network module, determining the positions of the targets, and cutting out the targets from the image data based on the positions to obtain multiple regions of interest; extracting image features of the regions of interest by using an image feature extraction neural network module to obtain first features; extracting text features of the label information of the target to be recognized by using a text feature extraction neural network module to obtain second features; merging the first features and the second features to obtain a merged matrix; interactively fusing the first features and the second features in the merged matrix by using a cross-modal feature fusion neural network module to obtain a fused matrix, and separating the fused matrix into two separated matrices; calculating a similarity matrix of the two separated matrices, and determining a target detection category according to the similarity matrix. 2.The cross-modal image object detection method using text information according to claim 1, characterized in that, The candidate box neural network module is a DETR target detection network. 3.The cross-modal image object detection method using text information according to claim 1, characterized in that, The image feature extraction neural network module is a VIT pre-training model. 4.The cross-modal image object detection method using text information according to claim 1, characterized in that, The text feature extraction neural network module is a BERT network. 5.The cross-modal image object detection method using text information according to claim 1, characterized in that, The cross-modal feature fusion neural network module comprises three layers of self-attention network layers and two layers of fully connected layers arranged in sequence, the three layers of self-attention network layers are used for interactively fusing the first features and the second features in the merged matrix to output a fused matrix; the two layers of fully connected layers are used for separating the fused matrix into two separated matrices.

6. A cross-modal image target detection device utilizing textual information, characterized in that, comprises: an acquisition module, configured to acquire image data and label information of a target to be recognized; a candidate box neural network module, configured to detect all targets in the image data, determine the positions of the targets, and cut out the targets from the image data based on the positions to obtain multiple regions of interest; an image feature extraction neural network module, configured to extract image features of the regions of interest to obtain first features; a text feature extraction neural network module, configured to extract text features of the label information of the target to be recognized to obtain second features; a merging module, configured to merge the first features and the second features to obtain a merged matrix; a cross-modal feature fusion neural network module, configured to interactively fuse the first features and the second features in the merged matrix to obtain a fused matrix, and separate the fused matrix into two separated matrices; a classification module, configured to calculate a similarity matrix of the two separated matrices, and determine a target detection category according to the similarity matrix. The cross-modal feature fusion neural network module comprises three layers of self-attention network layers and two layers of fully connected layers arranged in sequence, the three layers of self-attention network layers are used for interactively fusing the first features and the second features in the merged matrix to output a fused matrix; 7.The cross-modal image object detection apparatus using text information according to claim 6, characterized in that, the two layers of fully connected layers are used for separating the fused matrix into two separated matrices. ​ 8.The cross-modal image object detection apparatus using text information according to claim 6, characterized in that, The classification module comprises: a calculation unit configured to perform matrix multiplication on two separated matrices and perform Soft-Max normalization operation on the obtained result to obtain a similarity matrix; and a comparison unit configured to compare the similarity of labels in the similarity matrix with a threshold value, remove labels less than the threshold value, and take the remaining labels as the categories of target detection. 9.An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the cross-modal image target detection method using text information according to any one of claims 1-5 when executing the computer program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the cross-modal image target detection method using text information according to any one of claims 1-5.

Images