General visual target localization methods, devices, electronic equipment and storage media

Abstract

This invention relates to a universal visual target localization method, apparatus, electronic device, and storage medium, belonging to the field of image recognition technology. The universal visual target localization method includes: acquiring a target image containing the target object to be located and text information describing the target object; extracting linguistic features from the text information using a feature extraction module, and extracting visual features from the target image based on the linguistic features; decoding the linguistic and visual features using an adaptive cross-modal decoding module to obtain target query features of the target object; and locating the target object in the target image based on the target query features using a target localization module. This invention adjusts the feature mapping under the guidance of linguistic information when extracting visual features, enabling the model to more flexibly extract contextual information related to the target object, resulting in more accurate visual localization.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a universal visual target localization method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of image recognition technology, visual positioning technology is also being used more and more widely.

[0003] Visual localization aims to locate target objects in an image based on a given text description. It uses natural language to describe the attributes, location, and relationships of the target object, enabling it to adapt to complex scenarios that require collaborative understanding of images and language. Most existing methods fix the spatial and channel dimensions of features during the encoding process, which limits the model's ability to flexibly capture context related to the target object. At the same time, local information of key attributes of the target object is usually extracted using information from different modalities. Most existing methods rely on global information while ignoring local information containing key attributes of the target object, and cannot effectively fuse attribute information from different modalities, resulting in inaccurate extraction of key attributes of the target object.

[0004] This shows that existing visual positioning technologies cannot fully integrate information from different modalities, resulting in inaccurate visual positioning. Summary of the Invention

[0005] In view of this, it is necessary to provide a universal visual target localization method, device, electronic device and storage medium to solve the problem that existing visual localization technologies cannot fully integrate information between different modalities, resulting in inaccurate visual localization.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a universal visual target localization method, comprising:

[0007] Acquire the image to be identified, which contains the target object to be located, and the text information describing the target object to be located;

[0008] The feature extraction module is used to extract the linguistic features of the text information, and the visual features of the image to be recognized are extracted based on the linguistic features;

[0009] An adaptive cross-modal decoding module is used to decode linguistic and visual features to obtain the target query features of the target object to be located.

[0010] The target localization module is used to locate the target object in the image to be identified based on the target query features.

[0011] In one possible embodiment of the present invention, the feature extraction module includes a language-guided context encoding submodule, which extracts language features from text information and extracts visual features from the image to be recognized based on the language features, including:

[0012] A word segmenter is used to segment text information into words and convert them into text tags, and a pre-defined first neural network model is used to extract language features from the text tags;

[0013] A pre-defined second neural network model is used to extract primary visual features of the image to be identified, and a language-guided context encoding submodule is used to extract context visual features containing context features of the target object to be located based on language features and primary visual features.

[0014] Visual features are extracted from contextual visual features using a pre-defined third neural network model.

[0015] In one possible embodiment of the present invention, a language-guided context encoding submodule is used to extract contextual visual features containing contextual features of the target object to be located based on language features and primary visual features, including:

[0016] Primary visual features are mapped to a first vector through a fully connected layer, and linguistic features are mapped to a second vector through a fully connected layer. A multi-head cross-modal attention mechanism is used to fuse the first and second vectors to obtain preprocessed features. The formula for calculating the preprocessed features is as follows:

[0017]

[0018] in, For preprocessing features, The initial visual features are represented by the first vector. The second vector is and , For multi-head cross-modal attention mechanisms;

[0019] Based on the preprocessed features and the dimension vector of the preprocessed features, the contextual visual features containing the contextual features of the target object to be located are determined. The calculation formula for the visual features in the contextual visual features is as follows:

[0020]

[0021] in, For contextual visual features, Preprocessing features The first dimension vector, Preprocessing features The second dimension vector, Preprocessing features The third dimension vector, for The channel dimension.

[0022] In one possible embodiment of the present invention, the calculation formulas for the first dimension vector, the second dimension vector, and the third dimension vector are as follows:

[0023]

[0024]

[0025]

[0026] in, It is a fully connected layer used for expansion. The channel dimension, It contains two convolutional layers, one for scaling. Spatial dimension, one used for expansion The channel dimension, Convolutional layers are used for scaling. Spatial dimension.

[0027] In one possible embodiment of the present invention, the adaptive cross-modal decoding module has N layers. The i-th layer adaptive cross-modal decoding module is used to decode the language features and visual features to obtain the target query features of the target object to be located, including:

[0028] The visual features, target query features, and language features output by the (i-1)th layer adaptive cross-modal decoding module are used as input to the i-th layer adaptive cross-modal decoding module to update the visual features, obtaining intermediate visual features. The formula for calculating the intermediate visual features is as follows:

[0029]

[0030] in, As a mesoscopic visual feature, , and The vectors are obtained by mapping visual features through three fully connected layers, and M represents the selectable attention area of ​​the target object to be located. Let be the probability that the j-th type of attentionable range is selected. For vectors Channel dimension;

[0031] The language features and intermediate visual features are input into a multi-head cross-modal attention layer to fuse the language information and obtain updated visual features.

[0032] The updated visual features and the target query features output by the (i-1)th layer adaptive cross-modal decoding module are input into the multi-head cross-modal attention layer to obtain intermediate target query features;

[0033] The intermediate target query features are processed by a feedforward neural network to generate the target query features output by the i-th layer adaptive cross-modal decoding module, where the target query features output by the N-th layer adaptive cross-modal decoding module are the target query features of the target object to be located.

[0034] In one possible embodiment of the present invention, a target localization module locates the target object to be identified in the image based on target query features, including:

[0035] The target query features are reduced in dimensionality using an average pooling layer to obtain dimensionality-reduced features.

[0036] A fully connected layer is used to map and predict the dimensionality reduction features to obtain the bounding box of the target object in the image to be identified.

[0037] In one possible embodiment of the present invention, the target localization module further includes a loss function, which is:

[0038]

[0039]

[0040] Where L is the total loss function, Let the loss function be the loss function of the embankment. For the generalized intersection-union loss function, , and For weight parameters, The coordinates of the predicted bounding box of the target object to be located. The coordinates of the actual bounding box of the target object to be located. and The center point of the coordinate frame. and These are the width and height of the coordinate frame.

[0041] Secondly, the present invention also provides a universal visual target localization device, comprising:

[0042] The information acquisition module is used to acquire an image of the target object to be identified, which contains the target object to be located, and text information describing the target object to be located.

[0043] The feature extraction module is used to extract the linguistic features of the text information and extract the visual features of the image to be recognized based on the linguistic features.

[0044] The feature processing module is used to decode language features and visual features using an adaptive cross-modal decoding module to obtain the target query features of the target object to be located.

[0045] The localization module is used to locate the target object in the image to be identified based on the target query features, using the target localization module.

[0046] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein,

[0047] Memory, used to store programs;

[0048] A processor, coupled to a memory, is used to execute a program stored in the memory to implement the steps in the general visual target localization method of any of the above embodiments.

[0049] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps of the general visual target localization method of any of the above embodiments.

[0050] The beneficial effects of this invention are as follows: The general visual target localization method provided by this invention extracts the linguistic features of the text information using a feature extraction module, and extracts the visual features of the image to be identified based on the linguistic features. This achieves the purpose of guiding and adjusting the feature mapping based on linguistic information, enabling the model to extract contextual information related to the target object more flexibly. An adaptive cross-modal decoding module decodes the linguistic features and the visual features to obtain the target query features of the target object to be located, which can realize the fusion of features from different modalities and improve the accuracy of feature extraction. The target localization module locates the target object to be located in the image to be identified based on the target query features, making visual localization more accurate. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a general visual target localization method provided in an embodiment of the present invention;

[0053] Figure 2 A model structure diagram provided for an embodiment of the present invention;

[0054] Figure 3This is a flowchart illustrating a feature extraction method provided in an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of a visual feature extraction structure provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of an adaptive cross-modal decoding layer structure provided in an embodiment of the present invention;

[0057] Figure 6 This is a flowchart illustrating a coordinate frame positioning method provided in an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of the structure of a universal visual target localization device provided in an embodiment of the present invention;

[0059] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0060] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0061] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0062] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0063] A specific embodiment of the present invention, such as Figure 1 As shown, a general visual target localization method is disclosed, including:

[0064] S101, acquire the image to be identified containing the target object to be located and the text information used to describe the target object to be located.

[0065] In this embodiment of the invention, the provided general visual target localization method is implemented through a flexible context and adaptive attention visual localization model of a general visual dialogue. The structure of the flexible context and adaptive attention visual localization model of the general visual dialogue is as follows: Figure 2 As shown, it includes a feature extraction module, an adaptive cross-modal decoding module, and a target localization module. The working principle of each module will be explained in detail later in this invention.

[0066] In this embodiment of the invention, the image to be identified includes a target object to be located, and the text information is used to describe the attributes, location, and relationships of the target object to be located.

[0067] S102, the feature extraction module is used to extract the language features of the text information, and the visual features of the image to be recognized are extracted based on the language features.

[0068] In this embodiment of the invention, the feature extraction module is used to extract linguistic features from the text data and to extract visual features of the target image to be located in the image to be recognized using the linguistic features. The feature extraction module includes multiple neural network models, which are pre-trained to recognize the image and text information to obtain linguistic and visual features. The specific methods for extracting linguistic and visual features will be described in detail later in this invention.

[0069] S103 uses an adaptive cross-modal decoding module to decode language features and visual features to obtain the target query features of the target object to be located.

[0070] In this embodiment of the invention, the adaptive cross-modal decoding module can achieve the fusion decoding of language features and visual features, thereby determining the target query features of the target object to be located. When determining the target query features of the target object to be located, the scope of interest can be adaptively selected, which can not only query global information, but also capture local details containing the key attributes of the target object. The specific method for determining the target query features will be described in detail later in this invention.

[0071] S104, The target localization module locates the target object in the image to be identified based on the target query features.

[0072] In this embodiment of the invention, the target localization module is used to locate the target object to be located using target query features and to regress and predict the coordinate frame of the target object to be located. The specific localization process will be described in detail later in this invention.

[0073] The universal visual target localization method provided by this invention extracts linguistic features from text information using a feature extraction module, and then extracts visual features from the image to be identified based on these linguistic features. This achieves the goal of guiding and adjusting feature mapping based on linguistic information, enabling the model to extract contextual information related to the target object more flexibly. An adaptive cross-modal decoding module decodes the linguistic and visual features to obtain the target query features of the target object to be located, achieving feature fusion across different modalities and improving the accuracy of feature extraction. Finally, a target localization module locates the target object in the image to be identified based on the target query features, resulting in more accurate visual localization.

[0074] In some possible embodiments of the present invention, such as Figure 3 As shown, a feature extraction module is used to extract linguistic features from the text information, and visual features of the image to be recognized are extracted based on the linguistic features, including:

[0075] S301, a word segmenter is used to segment text information into words and convert them into text tags, and a preset first neural network model is used to extract language features from the text tags;

[0076] S302, a preset second neural network model is used to extract primary visual features of the image to be identified, and a language-guided context encoding submodule is used to extract context visual features containing context features of the target object to be located based on language features and primary visual features;

[0077] S303 uses a pre-defined third neural network model to extract visual features from contextual visual features.

[0078] In this embodiment of the invention, the feature extraction module segments the text information into text tags using a word segmenter, encodes the text tags according to the vocabulary, and standardizes their length. After processing, the text tags are input into the pre-trained BERT network to extract language features. In the embodiments of this application Set to 40, language features , The feature channel dimension is set to 256. The feature extraction module converts the image to be recognized into RGB format and uniformly resizes it. After normalizing the image data to a uniform size, it is input into a pre-trained ResNet-50 network to extract primary visual features. Primary visual features and language features The input language-guided context encoding submodule flexibly extracts visual features containing the context of the target object to be located. Visual features containing the context of the target object to be located Visual features are extracted from the pre-trained 6-layer Transformer encoding layer. In this embodiment, the feature size extracted by the pre-trained ResNet-50 network is [size missing]. Reshape it into ,in and If set to 20, then 400. Primary visual features. Visual features containing the context of the target object to be located Visual features The pre-trained ResNet-50 network and the initialization of the 6 Transformer coding layers are based on the parameters trained on the DETR network.

[0079] The embodiments of the present invention extract features from the image and text information to be identified, thereby obtaining more contextual information related to the target object, which is beneficial for locating the target object.

[0080] In some possible embodiments of the present invention, a language-guided context encoding submodule is used to extract contextual visual features containing contextual features of the target object to be located based on language features and primary visual features, including:

[0081] Primary visual features are mapped to a first vector through a fully connected layer, and linguistic features are mapped to a second vector through a fully connected layer. A multi-head cross-modal attention mechanism is used to fuse the first and second vectors to obtain preprocessed features. The formula for calculating the preprocessed features is as follows:

[0082]

[0083] in, For preprocessing features, The initial visual features are represented by the first vector. The second vector is and , For multi-head cross-modal attention mechanisms;

[0084] Based on the preprocessed features and the dimension vector of the preprocessed features, the contextual visual features containing the contextual features of the target object to be located are determined. The calculation formula for the visual features in the contextual visual features is as follows:

[0085]

[0086] in, For contextual visual features, Preprocessing features The first dimension vector, Preprocessing features The second dimension vector, Preprocessing features The third dimension vector, for The channel dimension.

[0087] In this embodiment of the invention, the language-guided context encoding submodule utilizes language information to guide and release the feature space and channel dimensions, so as to flexibly extract visual features containing the context of the target object. ,like Figure 4 As shown, primary visual features After being mapped to a vector by a fully connected layer linguistic features The vectors are mapped through two fully connected layers. , Preprocessed features are obtained by fusing visual and language features using a multi-head cross-modal attention mechanism. Preprocessing features The calculation formula is as follows:

[0088]

[0089] in, For multi-head cross-modal attention mechanisms;

[0090] Expanding preprocessing features using fully connected layers The channel dimension is used as a vector The calculation formula is as follows:

[0091]

[0092] in, It is a fully connected layer used for expansion. Channel dimension;

[0093] Scaling preprocessed features using convolutional layers Spatial dimensions and expanded preprocessing features The channel dimension is used as a vector The calculation formula is as follows:

[0094]

[0095] in, It contains two convolutional layers, one for scaling. Spatial dimension, one used for expansion Channel dimension;

[0096] Scaling preprocessed features using convolutional layers Spatial dimension as a vector The calculation formula is as follows:

[0097]

[0098] in, Convolutional layers are used for scaling. Spatial dimension;

[0099] Using vectors , , Calculate visual features containing the context of the target object. and guarantee and Consistent size ensures that visual features can be extracted in subsequent input 6 Transformer coding layers. , The calculation formula is as follows:

[0100]

[0101] in, for The channel dimension.

[0102] In this embodiment of the invention, the preprocessing features , , , Set the channel dimension to 1.25 times the original. Set the spatial dimension to 320; set the spatial dimension back to the original value. times, It is 100. (Using vectors) For example, vector Preprocessing features The preprocessed features are obtained after two convolutional layers. Remodeling of Form, through a The size of the convolutional kernel compresses the spatial dimension, transforming the features into... And then through a The convolution kernel maps it to Finally, the features are reconstructed to obtain The above is just an example; the specific channel and spatial dimensions after scaling can be set as needed. Although the spatial and channel dimensions change during the calculation, the final features obtained remain the same. The dimensions remain unchanged, which facilitates subsequent calculations.

[0103] In the embodiments of the present invention, when extracting visual features, the feature mapping is adjusted under the guidance of linguistic information, while releasing the spatial and channel dimensions of the features, so that the model can extract contextual information related to the target object more flexibly.

[0104] In some possible embodiments of the present invention, the adaptive cross-modal decoding module has N Layer, using the first i The layer-adaptive cross-modal decoding module decodes linguistic and visual features to obtain the target query features of the target object to be located, including:

[0105] The first i The visual features, target query features, and linguistic features output by the -1 layer adaptive cross-modal decoding module are used as the first... i The input to the layer-adaptive cross-modal decoding module updates the visual features to obtain intermediate visual features. The formula for calculating the intermediate visual features is as follows:

[0106]

[0107] in, As a mesoscopic visual feature, , and The vectors are obtained by mapping visual features through three fully connected layers. M The selectable area of ​​interest for the target object to be located. Let be the probability that the j-th type of attentionable range is selected. For vectors Channel dimension;

[0108] The language features and intermediate visual features are input into a multi-head cross-modal attention layer to fuse the language information and obtain updated visual features.

[0109] Updated visual features and the first i The target query features output from the -1 layer adaptive cross-modal decoding module are input into the multi-head cross-modal attention layer to obtain intermediate target query features;

[0110] The intermediate target query features are processed by a feedforward neural network to generate the first... i The target query features output by the layer adaptive cross-modal decoding module, where the first... N The target query features output by the layer adaptive cross-modal decoding module are the target query features of the target object to be located.

[0111] In this embodiment of the invention, the adaptive cross-modal decoding module consists of a layer of adaptive cross-modal decoding layers, such as... Figure 5 As shown, the first The specific steps of the layer-adaptive cross-modal decoding layer are as follows: language features , No. Visual features output by the layer adaptive decoding layer and target query features As the first The layer adapts to the input of the decoding layer; it converts visual features The vectors are mapped through three fully connected layers. , , Calculate attention score :

[0112]

[0113] Assuming there is a total Selectable areas of interest, with a mask matrix set for each area of ​​interest. mask matrix With attention score The same size, and the same as the attention score. Perform multiplication at corresponding positions to obtain the mask matrix. Each row indicates which elements that row's element is following; a value of 1 indicates following, and 0 indicates not following. The settings... The scope of attention is each element in spatial location centered on itself. , The range of matrices of size ..., the first This is a normal attention mechanism to ensure that global information is not lost; visual features The probability of each area of ​​interest being selected is predicted using an average pooling layer and a fully connected layer. The calculation formula is as follows:

[0114]

[0115] in Indicates the average pooling layer. Indicates a fully connected layer;

[0116] Expand the selection of attention ranges and update visual features using the probability of each attention range being selected. Obtain intermediate visual features This makes it a continuous and differentiable operation, and the calculation formula is as follows:

[0117]

[0118] in For the first The probability of a focus area being selected. It is a vector Channel dimension;

[0119] Language features and intermediate visual features The updated visual features are obtained by fusing language information into a multi-head cross-modal attention layer. ; target query features and updated visual features In the multi-head cross-modal attention layer, visual features related to the target object are used to obtain intermediate target query features. ; Query features of intermediate targets New target query features are generated using a feedforward neural network. ;when hour, = , ;when At that time, the target query features generated by the adaptive cross-modal decoding module are obtained. .

[0120] In this embodiment of the invention, the first Visual features output by the layer adaptive decoding layer Target query features mask matrix Attention score The adaptive decoding module has a total of 6 adaptive decoding layers, namely... ;Set as follows , , And the four selectable attention ranges of normal global self-attention, namely .

[0121] In the process of updating target query features, the embodiments of the present invention adaptively select the scope of interest, which can not only query global information, but also capture local details containing key attributes of the target object, which is beneficial to improving the model's localization ability in complex scenarios that require detailed understanding.

[0122] In some possible embodiments of the present invention, such as Figure 6 As shown, the target localization module locates the target object in the image to be identified based on the target query features, including:

[0123] S601, the average pooling layer is used to reduce the dimensionality of the target query features to obtain the dimensionality-reduced features;

[0124] S602 uses a fully connected layer to map and predict the dimensionality reduction features, thereby obtaining the bounding box of the target object in the image to be identified.

[0125] In this embodiment of the invention, the target localization module utilizes target query features. Locate the target object and predict its coordinate frame using regression. In this embodiment, the target query features output by the target localization module First, dimensionality reduction is performed using average pooling. The resulting dimensionality-reduced features Finally, a fully connected layer is used to map and predict the coordinate box of the target object. , ,in The coordinates of the center point of the coordinate frame. These are the width and height of the coordinate frame.

[0126] In this embodiment of the invention, the target localization module further includes a loss function, which is:

[0127]

[0128]

[0129] in, L For the total loss function, Let the loss function be the loss function of the embankment. For the generalized intersection-union loss function, , and For weight parameters, The coordinates of the predicted bounding box of the target object to be located. The coordinates of the actual bounding box of the target object to be located. and The center point of the coordinate frame. and These are the width and height of the coordinate frame.

[0130] In this embodiment of the invention, when training a general visual dialogue flexible context and adaptive attention visual localization model, the target object's real bounding box is used. and prediction boxes The total loss function value is calculated, backpropagation is performed, and the connection weights are optimized through the selected optimizer and corresponding parameters. The model with the highest prediction accuracy after multiple rounds of training is the final general visual dialogue flexible context and adaptive attention visual localization method.

[0131] In this embodiment of the invention, the total loss function value of the model includes the GIoU (Generalized Intersection over Union) loss function. and point-edge loss function The two-part model's total loss function The calculation formula is as follows:

[0132]

[0133] in, This represents the value of the weight parameter.

[0134] Furthermore, the point-edge loss function The calculation formula is:

[0135]

[0136] in, and Represents the weight parameters. The coordinates of the predicted bounding box of the target object. The coordinates of the target object's true bounding box. and The center point of the coordinate frame. and The width and height of the coordinate frame are, in this embodiment... , , The values ​​are 5, 0.5, and 2 respectively.

[0137] This invention introduces a point-edge loss function for the coordinate frame to calculate the absolute loss of the center point of the coordinate frame and the relative loss of the side length of the coordinate frame, thereby alleviating the training bias caused by uneven distribution of coordinate frame size.

[0138] To better implement the general visual target localization method in the embodiments of the present invention, based on the general visual target localization method, correspondingly, as follows: Figure 7 As shown, this embodiment of the invention also provides a universal visual target localization device, the universal visual target localization device 700 including:

[0139] The information acquisition module 701 is used to acquire an image to be identified containing the target object to be located and text information describing the target object to be located.

[0140] The feature extraction module 702 is used to extract the language features of the text information and extract the visual features of the image to be recognized based on the language features.

[0141] The feature processing module 703 is used to decode the language features and visual features using the adaptive cross-modal decoding module to obtain the target query features of the target object to be located.

[0142] The positioning module 704 is used to locate the target object in the image to be identified based on the target query features by the target positioning module.

[0143] The universal visual target localization device 700 provided in the above embodiments can realize the technical solutions described in the embodiments of the universal visual target localization method. The specific implementation principles of each module or unit can be found in the corresponding content in the embodiments of the universal visual target localization method, and will not be repeated here.

[0144] This invention adjusts the mapping of visual features by guiding them with linguistic features, while simultaneously releasing the spatial and channel dimensions of the features. This improves the flexibility of the model in extracting features and obtaining more contextual information related to the target object, thus facilitating the localization of the target object. During the process of updating the target query features, the scope of interest is adaptively selected, which not only allows for the querying of global information but also captures local details containing key attributes of the target object. This is beneficial for improving the model's localization ability in complex scenes that require detailed understanding. Considering the influence of coordinate frames of different sizes, a point-edge loss function for the coordinate frame is introduced to calculate the absolute loss of the center point of the coordinate frame and the relative loss of the side length of the coordinate frame, thereby mitigating the training bias caused by uneven distribution of coordinate frame sizes.

[0145] like Figure 8 As shown, the present invention also provides an electronic device 800. The electronic device 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the electronic device 800 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0146] In some embodiments, processor 801 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 802 or process data, such as the general visual target localization method of the present invention.

[0147] In some embodiments, processor 801 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 801 may be local or remote. In some embodiments, processor 801 may be implemented on a cloud platform. In some embodiments, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0148] In some embodiments, memory 802 may be an internal storage unit of electronic device 800, such as a hard disk or memory of electronic device 800. In other embodiments, memory 802 may also be an external storage device of electronic device 800, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 800.

[0149] Furthermore, the memory 802 may include both internal storage units of the electronic device 800 and external storage devices. The memory 802 is used to store application software and various types of data installed on the electronic device 800.

[0150] In some embodiments, display 803 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 803 is used to display information from electronic device 800 and to display a visual user interface. Components 801-803 of electronic device 800 communicate with each other via a system bus.

[0151] In some embodiments, when the processor 801 executes the target location program in the memory 802, the following steps may be performed:

[0152] Acquire the image to be identified, which contains the target object to be located, and the text information describing the target object to be located;

[0153] The feature extraction module is used to extract the linguistic features of the text information, and the visual features of the image to be recognized are extracted based on the linguistic features;

[0154] An adaptive cross-modal decoding module is used to decode linguistic and visual features to obtain the target query features of the target object to be located.

[0155] The target localization module is used to locate the target object in the image to be identified based on the target query features.

[0156] It should be understood that when the processor 801 executes the target location program in the memory 802, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0157] Accordingly, this application also provides a computer-readable storage medium for storing a computer-readable program or instruction. When the program or instruction is executed by a processor, it can implement the steps or functions of the general visual target localization method provided in the above-described method embodiments.

[0158] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0159] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A general visual target positioning method, characterized in that, The method comprises the following steps: obtaining a to-be-recognized image containing a to-be-positioned target object and text information used for describing the to-be-positioned target object; extracting language features of the text information by using a feature extraction module, and extracting visual features of the to-be-recognized image based on the language features; performing multi-scale adaptive decoding on the language features and the visual features by using an adaptive cross-modal decoding module to obtain target query features of the to-be-positioned target object; positioning the to-be-positioned target object in the to-be-recognized image based on the target query features by using a target positioning module; the feature extraction module comprises a language-guided context encoding submodule, and the language features of the text information are extracted by using the feature extraction module, and the visual features of the to-be-recognized image are extracted based on the language features, which comprises the following steps: converting word segmentation of the text information into text tokens by using a word segmenter, and extracting language features in the text tokens by using a preset first neural network model; extracting primary visual features of the to-be-recognized image by using a preset second neural network model, and extracting context visual features containing context features of the to-be-positioned target object based on the language features and the primary visual features by using the language-guided context encoding submodule; extracting visual features in the context visual features by using a preset third neural network model; The adaptive cross-modal decoding module has N layer, adopts the first i The adaptive cross-modal decoding module performs multi-scale adaptive decoding on the language features and the visual features to obtain target query features of the target object to be positioned. The first i layer adaptive cross-modal decoding module outputs a visual feature, a target query feature and the language feature as inputs of the second i layer adaptive cross-modal decoding module, updates the visual feature to obtain an intermediate visual feature, and a calculation formula of the intermediate visual feature is as follows: wherein, is the intermediate visual feature, , and is the vector mapped by the three fully connected layers respectively from the visual feature, M is the selected attention range of the target object to be located, is the selected probability of the jth attention range, is the channel dimension of the vector . fusing language information in a multi-head cross-modal attention layer by inputting the language features and the intermediate visual features to obtain updated visual features; inputting the updated visual feature and the first target query feature into a multi-head cross-modal attention layer to obtain an intermediate target query feature i inputting the target query feature output by the 1st layer adaptive cross-modal decoding module into a multi-head cross-modal attention layer to obtain an intermediate target query feature The intermediate target query feature is input into a feedforward neural network to generate a first target query feature i The target query feature output by the layer-adaptive cross-modal decoding module is used as the target query feature of the to-be-positioned target object. N The target query feature output by the layer-adaptive cross-modal decoding module is used as the target query feature of the to-be-positioned target object.

2. The universal visual target positioning method of claim 1, wherein, the language-guided context encoding submodule extracts context visual features containing context features of the to-be-positioned target object based on the language features and the primary visual features, which comprises the following steps: mapping the primary visual features into a first vector through a fully connected layer, mapping the language features into a second vector through a fully connected layer, and fusing the first vector and the second vector to obtain preprocessed features by using a multi-head cross-modal attention mechanism, wherein a calculation formula of the preprocessed features is: wherein, is a pre-processing feature, is a primary visual feature, the first vector is , the second vector is and , is a multi-head cross-modal attention mechanism; determining context visual features containing context features of the to-be-positioned target object based on the preprocessed features and a dimension vector of the preprocessed features, wherein a calculation formula of visual features in the context visual features is: wherein, is a contextual visual feature, is a pre-processed feature is a first dimension vector of the pre-processed feature is a pre-processed feature is a second dimension vector of the pre-processed feature is a pre-processed feature is a third dimension vector of the pre-processed feature is a channel dimension of .

3. The universal visual target positioning method of claim 2, wherein, calculation formulas of the first dimension vector, the second dimension vector and the third dimension vector are: wherein, is a fully connected layer for expanding the channel dimension of , contains two convolutional layers, one for scaling the spatial dimension of and one for expanding the channel dimension of , is a convolutional layer for scaling the spatial dimension of .

4. The universal visual target positioning method of claim 1, wherein, the target positioning module positions the to-be-positioned target object in the to-be-recognized image based on the target query features, which comprises the following steps: performing dimension reduction on the target query features by using an average pooling layer to obtain reduced dimension features; mapping and predicting the reduced dimension features by using a fully connected layer to obtain a coordinate frame of the to-be-positioned target object in the to-be-recognized image.

5. The universal visual target positioning method of claim 1, wherein, The target positioning module further comprises a loss function, and the loss function is: wherein L is a total loss function, is a bank side loss function, is a generalized intersection over union loss function, , and are weight parameters, is a predicted bounding box coordinate of a target object to be positioned, is a real bounding box coordinate of a target object to be positioned, and are center points of the coordinate box, and are width and height of the coordinate box.

6. A general visual target positioning apparatus for performing the general visual target positioning method according to any one of claims 1 to 5, characterized by, The method comprises the following steps: an information acquisition module is configured to obtain a to-be-recognized image containing a to-be-positioned target object and text information used for describing the to-be-positioned target object; a feature extraction module is configured to extract language features of the text information by using a feature extraction module, and extract visual features of the to-be-recognized image based on the language features; The feature processing module is configured to perform multi-scale adaptive decoding on the language feature and the visual feature by using the adaptive cross-modal decoding module to obtain a target query feature of the target object to be positioned. The positioning module is configured to position the target object to be positioned in the image to be recognized based on the target query feature by using a target positioning module.

7. An electronic device, comprising: comprising a memory and a processor, wherein, The memory is configured to store a program. The processor, coupled with the memory, is configured to execute the program stored in the memory to implement the steps in the general visual target positioning method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer readable program or instruction is stored, and the program or instruction is executed by a processor to implement the steps in the general visual target positioning method according to any one of claims 1 to 5.

Images