Target recognition method and model thereof, electronic device, and storage medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a target recognition method and model thereof, an electronic device and a storage medium. In the target recognition method, a target visual image and representative information are first acquired, an image feature vector is formed according to the target visual image, a text feature vector is formed according to the representative information, further, the image feature vector and the text feature vector are subjected to correlation mapping processing to generate a multi-modal feature sequence, then, semantic similarity initial query is performed in the multi-modal feature sequence based on the text feature vector to obtain a feature description vector corresponding to a target object, and finally, a target anchor box for recognizing the target object is generated in the target visual image based on the feature description vector and the image feature vector, so that the object described by the representative information can be accurately positioned in the visual image, and the accuracy of target recognition is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a target recognition method and model, electronic device, and storage medium. Background Technology

[0002] Referring Expression Comprehension (REC) is the task of locating objects described by natural language in an image. The text refers to a single object or a group of objects in the image, and the goal is to detect the region containing the target object and place a bounding box. It should be understood that referring expression refers to the natural language expression describing a specific object in a scene, and referring expression comprehension requires the algorithm to locate the object described by the referring expression in a given image. Referring expression comprehension tasks provide value for many real-world applications, such as robotics, image editing, and public safety monitoring. In related technologies, when calculating the correlation between natural language expressions and visual images, the position of multimodal context in the image is determined by pure semantic similarity, making it difficult to achieve accurate localization of objects described by referring expressions in visual images. Therefore, how to achieve relatively accurate localization of objects described by referring expressions in visual images has become a major problem that urgently needs to be solved in the industry. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a target recognition method and its model, electronic device, and storage medium, which can achieve relatively accurate positioning of the object described by the pointer in a visual image.

[0004] The target recognition method according to the first aspect of this application includes:

[0005] Acquire a target visual image and pointer expression information, wherein the pointer expression information is used to describe the target object in the target visual image;

[0006] An image feature vector is formed based on the target visual image, and a text feature vector is formed based on the index expression information;

[0007] The image feature vector and the text feature vector are subjected to correlation mapping processing to generate a multimodal feature sequence;

[0008] Based on the text feature vector, an initial semantic similarity query is performed in the multimodal feature sequence to obtain the feature description vector corresponding to the target object;

[0009] Based on the feature description vector and the image feature vector, a target anchor box for identifying the target object is generated in the target visual image.

[0010] According to some embodiments of this application, the step of performing an initial semantic similarity query on the multimodal feature sequence based on the text feature vector to obtain a feature description vector corresponding to the target object includes:

[0011] A prototype decoder is constructed based on a cross-attention mechanism, and the prototype decoder includes a cross-attention layer;

[0012] The text feature vector is used as the initial query request, and the semantic similarity initial query is performed on the multimodal feature sequence through the cross-attention layer to obtain the feature description vector.

[0013] According to some embodiments of this application, generating a target anchor box for identifying the target object in the target visual image based on the feature description vector and the image feature vector includes:

[0014] Based on the feature description vector and the image feature vector, the anchor frame position information is obtained;

[0015] Based on the feature description vector, the anchor box position information, and the image feature vector, a semantic similarity optimization query is performed in the multimodal feature sequence to iteratively update the feature description vector and the anchor box position information;

[0016] When the iterative update meets the preset conditions, the target anchor frame is generated in the target visual image based on the anchor frame position information after the iterative update.

[0017] According to some embodiments of this application, the step of performing semantic similarity optimization query on the multimodal feature sequence based on the feature description vector, the anchor box position information, and the image feature vector, to iteratively update the feature description vector and the anchor box position information, includes:

[0018] In each round of the iterative update, the semantic similarity optimization query is performed on the multimodal feature sequence based on the feature description vector to update the feature description vector, and the anchor box position information is updated based on the image feature vector and the updated feature description vector.

[0019] According to some embodiments of this application, updating the anchor box position information based on the image feature vector and the updated feature description vector includes:

[0020] Based on the image feature vector and the updated feature description vector, the optimized anchor box information is obtained;

[0021] The optimized anchor frame information is compared with the feature description vector to obtain the predicted offset;

[0022] The anchor frame position information is updated based on the predicted offset.

[0023] According to some embodiments of this application, the step of forming an image feature vector based on the target visual image and forming a text feature vector based on the index expression information includes:

[0024] The target visual image is subjected to image patch encoding to obtain the image feature vector;

[0025] Semantic features are extracted from the indicated information to obtain the text feature vector.

[0026] According to some embodiments of this application, the step of performing correlation mapping processing on the image feature vector and the text feature vector to generate a multimodal feature sequence includes:

[0027] A multimodal encoder is constructed based on a self-attention mechanism, and the multimodal encoder includes a multi-head self-attention layer;

[0028] Feature association information is established based on the image feature vector and the text feature vector through the multi-head self-attention layer;

[0029] Based on the feature association information, the image feature vector and the text feature vector are concatenated to form the multimodal feature sequence.

[0030] The target recognition model according to the second aspect of this application includes:

[0031] The data acquisition module is used to acquire target visual images and pointer expression information, wherein the pointer expression information is used to describe the target object in the target visual image;

[0032] A multimodal encoder is used to form an image feature vector based on the target visual image, and to form a text feature vector based on the index expression information, and to perform correlation mapping processing on the image feature vector and the text feature vector to generate a multimodal feature sequence;

[0033] The prototype decoder is used to perform an initial semantic similarity query on the multimodal feature sequence based on the text feature vector to obtain a feature description vector corresponding to the target object;

[0034] An anchor frame decoder is used to generate target anchor frames in the target visual image for identifying the target object based on the feature description vector and the image feature vector.

[0035] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the target recognition method as described in any one of the embodiments of the first aspect of this application.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that is executed by a processor to implement the target recognition method as described in any one of the embodiments of the first aspect of this application.

[0037] The target recognition method, model, electronic device, and storage medium according to the embodiments of this application have at least the following beneficial effects:

[0038] In this target recognition method, it is necessary to first acquire a target visual image and index expression information. The index expression information is used to describe the target object in the target visual image. Then, an image feature vector is formed based on the target visual image, and a text feature vector is formed based on the index expression information. Further, correlation mapping processing is performed on the image feature vector and the text feature vector to generate a multimodal feature sequence. Then, based on the text feature vector, an initial semantic similarity query is performed in the multimodal feature sequence to obtain the feature description vector corresponding to the target object. Finally, based on the feature description vector and the image feature vector, a target anchor box for identifying the target object is generated in the target visual image. This target recognition method performs an initial semantic similarity query in the multimodal feature sequence based on the text feature vector to obtain the feature description vector corresponding to the target object, and finally generates a target anchor box for identifying the target object in the target visual image based on the feature description vector and the image feature vector. This enables relatively accurate localization of the object described by the index expression in the visual image, thereby improving the accuracy of target recognition.

[0039] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0040] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0041] Figure 1 This is a schematic diagram of the target recognition method provided in the embodiments of this application;

[0042] Figure 2This is another schematic diagram of the target recognition method provided in the embodiments of this application;

[0043] Figure 3 This is another schematic diagram of the target recognition method provided in the embodiments of this application;

[0044] Figure 4 This is another schematic diagram of the target recognition method provided in the embodiments of this application;

[0045] Figure 5 This is another schematic diagram of the target recognition method provided in the embodiments of this application;

[0046] Figure 6 This is another schematic diagram of the target recognition method provided in the embodiments of this application;

[0047] Figure 7 This is a schematic diagram of the modules of the target recognition model provided in the embodiments of this application;

[0048] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0050] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0051] In the description of this application, it should be understood that the orientation descriptions, such as up, down, left, right, front, and back, are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0052] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0053] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly. Those skilled in the art can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution. Furthermore, the identification of specific steps in the following text does not imply a limitation on the order of steps or execution logic. The execution order and logic between each step should be understood and inferred from the content described in the embodiments.

[0054] Attention mechanisms mimic the internal processes of biological observation, aligning internal experience with external perception to increase the precision of observation in specific regions. They are widely used in natural language processing tasks, particularly machine translation, due to their ability to quickly extract important features from sparse data. Self-attention and cross-attention are improvements on attention mechanisms, reducing reliance on external information and excelling at capturing the internal correlations of data or features. Self-attention addresses situations where neural networks receive inputs of many vectors of varying sizes with relationships between them, but training fails to fully leverage these relationships, resulting in poor model performance. It addresses the problem of fully connected neural networks failing to establish correlations among multiple related inputs by guiding the machine to notice the correlations between different parts of the overall input. Cross-attention, on the other hand, captures local information within image patches without alternating attention across the entire image, and applies attention between image patches partitioned from single-channel feature maps to capture global information. Therefore, cross-attention allows for performance maintenance with lower computational cost.

[0055] Referring Expression Comprehension (REC) is the task of locating objects described by natural language in an image. The text refers to a single object or a group of objects in the image, and the goal is to detect the region containing the target object and place a bounding box. It should be understood that referring expression refers to the natural language expression describing a specific object in a scene, and referring expression comprehension requires the algorithm to locate the object described by the referring expression in a given image. Referring expression comprehension tasks provide value for many real-world applications, such as robotics, image editing, and public safety monitoring. In related technologies, when calculating the correlation between natural language expressions and visual images, the position of multimodal context in the image is determined by pure semantic similarity, making it difficult to achieve accurate localization of objects described by referring expressions in visual images. Therefore, how to achieve relatively accurate localization of objects described by referring expressions in visual images has become a major problem that urgently needs to be solved in the industry.

[0056] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a target recognition method and its model, electronic device, and storage medium, which can achieve relatively accurate positioning of the object described by the pointer in a visual image.

[0057] Reference Figure 1 The target recognition method according to the first aspect of this application includes:

[0058] Step S101: Obtain the target visual image and the pointer expression information. The pointer expression information is used to describe the target object in the target visual image.

[0059] According to some embodiments provided in this application, the task of instruction expression understanding specifically refers to the process of locating the region where a target object is located in a target visual image based on instruction expression information. It can be clearly defined that the target visual image refers to an image containing the target object during the instruction expression understanding process, while the instruction expression information refers to information used to provide natural language instruction expression. According to some embodiments provided in this application, the textual semantic information in the instruction expression information directly or indirectly refers to the target object in the target visual image.

[0060] Step S102: Form an image feature vector based on the target visual image, and form a text feature vector based on the index expression information;

[0061] Step S103: Perform correlation mapping processing on the image feature vector and the text feature vector to generate a multimodal feature sequence;

[0062] According to some embodiments provided in this application, after acquiring the target visual image, it is necessary to segment the target visual image and extract image features through an image feature network to obtain an image feature vector. In some more specific embodiments, the image feature network can be a hierarchical model containing 12 layers of Swin Transformer sub-modules. Each layer of Swin Transformer sub-module is used to reduce the resolution of the input feature map and expand the receptive field layer by layer, thereby obtaining an image feature vector of a fixed size. According to some embodiments provided in this application, after acquiring the index expression information, it is necessary to perform text segmentation and text feature extraction on the index expression information through a natural language network to form a text feature vector. It should be noted that the text feature vector reflects the text semantic information in the index expression information. In some more specific embodiments of this application, the natural language network can be a trained semantic recognition model, wherein the natural language network can include 12 layers of BERT Transformer sub-modules. After extracting semantic features from the index expression information layer by layer, a feature vector of fixed length for each word is obtained. It should be understood that after obtaining the image feature vector and the text feature vector, the two unimodal feature sequences can be concatenated together to generate a multimodal feature sequence. It should be noted that in some embodiments of this application, during the process of generating a multimodal feature sequence by performing correlation mapping processing on image feature vectors and text feature vectors, it is necessary to first establish feature association information between the image feature vectors and text feature vectors. Then, based on the feature association information, the image feature vectors and text feature vectors are concatenated to obtain the multimodal feature sequence. It should be understood that the feature association information reflects the semantic correlation between the image feature vectors and text feature vectors, that is, the autocorrelation of the multimodal sequence. In some specific embodiments of this application, a multimodal encoder can be pre-constructed based on a self-attention mechanism. The multimodal encoder can be composed of 6 Transformer sub-modules. Each sub-module contains a multi-head self-attention layer and a forward propagation network layer. The self-attention layer calculates the autocorrelation of the multimodal sequence in multiple subspaces to form feature association information, thereby facilitating the concatenation of image feature vectors and text feature vectors to ultimately generate a multimodal feature sequence. The forward propagation layer can use the rectified linear unit (ReLU) as the activation function of the neuron. It is composed of two layers of linear transformation and ReLU nonlinear transformation, which can further enhance the expressive power of the multimodal encoder. It should be noted that further enhancing the expressive power of the multimodal encoder specifically means that the forward propagation layer can perform dimensionality upscaling or dimensionality reduction processing on single-modal feature sequences such as image feature vectors and text feature vectors, or multimodal sequences, so that the multimodal feature sequences have more expressive space.

[0063] According to some exemplary embodiments of this application, this application can encode multimodal feature sequences by mapping them to a two-dimensional sinusoidal embedding space. For image feature vectors, a two-dimensional sinusoidal encoder can be used to encode the position of each image block, mapping the image feature vectors to the two-dimensional sinusoidal embedding space. For text feature vector encoding, a multilayer perceptron can be used to convert the text feature vectors into two-dimensional text coordinates using the semantic information of the text words and the position of the text words in the sequence as two dimensions. Then, a two-dimensional sinusoidal encoder is used to encode the two-dimensional text coordinates, mapping them to the two-dimensional sinusoidal embedding space. Finally, the semantic features and positions of the multimodal sequence are embedded together into the multimodal encoder for multimodal information fusion, resulting in a multimodal feature sequence mapped to the two-dimensional sinusoidal embedding space.

[0064] Step S104: Based on the text feature vector, perform an initial semantic similarity query in the multimodal feature sequence to obtain the feature description vector corresponding to the target object;

[0065] Step S105: Based on the feature description vector and the image feature vector, generate a target anchor box in the target visual image for recognizing the target object.

[0066] It should be noted that the feature description vector refers to the image feature vector that actually establishes a semantic correlation mapping with the text feature vector. According to some exemplary embodiments of this application, an image semantic feature vector can be extracted from the image feature vector. The image semantic feature vector reflects the semantic information of the image. Since the text feature vector reflects the semantic information in the expressed information, a semantic correlation mapping can be established based on the semantic similarity between the image semantic feature vector and the text feature vector. The image semantic feature vector that actually establishes a semantic correlation mapping with the text feature vector is the feature description vector. According to some corresponding embodiments provided in this application, after performing correlation mapping processing on the image feature vector and the text feature vector to generate a multimodal feature sequence, a semantic correlation mapping is immediately established between the image feature vector and the text feature vector. That is, if an initial semantic similarity query is initiated in the multimodal feature sequence using the text feature vector, since a semantic correlation mapping is established between the image feature vector and the text feature vector, a feature description vector corresponding to the target object can be obtained based on the text feature vector. Furthermore, based on the feature description vector and the image feature vector, the region where the target object is located in the target visual image can be locked, thereby generating a target anchor box for identifying the target object. In some embodiments of this application, after obtaining the feature description vector and the image feature vector, the target visual image can be reconstructed according to each image feature vector, and the anchor frame can be generated by projecting the feature description vector onto the target visual image. The anchor frame can contain four dimensions, for example, x and y represent the center point of the anchor frame, and w and h represent the width and height of the anchor frame.

[0067] In this target recognition method, it is necessary to first acquire a target visual image and index expression information. The index expression information is used to describe the target object in the target visual image. Then, an image feature vector is formed based on the target visual image, and a text feature vector is formed based on the index expression information. Further, correlation mapping processing is performed on the image feature vector and the text feature vector to generate a multimodal feature sequence. Then, based on the text feature vector, an initial semantic similarity query is performed in the multimodal feature sequence to obtain the feature description vector corresponding to the target object. Finally, based on the feature description vector and the image feature vector, a target anchor box for identifying the target object is generated in the target visual image. This target recognition method performs an initial semantic similarity query in the multimodal feature sequence based on the text feature vector to obtain the feature description vector corresponding to the target object, and finally generates a target anchor box for identifying the target object in the target visual image based on the feature description vector and the image feature vector. This enables relatively accurate localization of the object described by the index expression in the visual image, thereby improving the accuracy of target recognition.

[0068] Reference Figure 2According to some embodiments of this application, an initial semantic similarity query is performed on a multimodal feature sequence based on text feature vectors to obtain a feature description vector corresponding to the target object, including:

[0069] Step S201: Construct a prototype decoder based on the cross-attention mechanism. The prototype decoder includes a cross-attention layer.

[0070] Step S202: The text feature vector is used as the initial query request. The semantic similarity is initially queried in the multimodal feature sequence through the cross-attention layer to obtain the feature description vector.

[0071] According to some embodiments provided in this application, self-attention and cross-attention are improvements on attention mechanisms, reducing reliance on external information and being better at capturing the internal correlations of data or features. Self-attention addresses the situation where neural networks receive many vectors of varying sizes as input, with certain relationships between them. However, during actual training, these relationships cannot be fully utilized, leading to poor model training results. Self-attention addresses the problem that fully connected neural networks cannot establish correlations for multiple related inputs, essentially aiming to make the machine notice the correlations between different parts of the overall input. Furthermore, cross-attention can capture local information within image patches without alternating attention across the entire image, and applies attention between image patches divided from single-channel feature maps to capture global information. Therefore, cross-attention can maintain performance with lower computational cost. Specifically, self-attention mechanisms provide an effective modeling method for capturing global contextual information through triplets consisting of a query, a key, and a value. In self-attention mechanisms, the data to be compared is often used as the query, and then aligned (i.e., similarity calculation) with each key-value pair based on the query. The obtained similarity is then normalized (e.g., using the Softmax function), and the comparison result is obtained based on the normalized calculation result. It should be noted that in cross-attention mechanisms, the query and key serve as different source features to perform initial semantic similarity queries in the multimodal feature sequence, obtaining feature description vectors, thereby achieving cross-modal relevance calculation.

[0072] In some exemplary embodiments of this application, the text feature vector to be compared is used as the initial query request, and then the semantic similarity initial query is initiated based on the initial query request to align with each key-value pair in the multimodal feature sequence (i.e., similarity calculation). In the multimodal feature sequence, the image feature vector can be used as the key-value pair, and then the calculated similarity is normalized to obtain the feature description vector based on the normalized calculation result.

[0073] According to some specific embodiments provided in this application, the prototype decoder may include a cross-attention layer and a weighted average pooling layer. In the cross-attention layer, the text feature vector to be compared is used as the initial query request to calculate similarity with each key-value pair in the multimodal feature sequence, where each image feature vector is used as the key-value pair. It should be noted that semantic similarity calculation with each image feature vector pair in the multimodal feature sequence based on the text feature vector can be achieved in various ways, including, but not limited to, using dot products, weights, concatenation weights, perceptrons, etc. According to some embodiments of this application, after dimensionality reduction processing of the feature description vector and image feature vector by the weighted average pooling layer, the target visual image is reconstructed according to each image feature vector, and anchor boxes are generated by projecting the feature description vectors onto the target visual image. The anchor boxes may contain four dimensions, for example, x and y represent the center point of the anchor box, and w and h represent the width and height of the anchor box.

[0074] Reference Figure 3 According to some embodiments of this application, a target anchor box for identifying target objects is generated in a target visual image based on feature description vectors and image feature vectors, including:

[0075] Step S301: Obtain anchor box position information based on feature description vector and image feature vector;

[0076] Step S302: Based on the feature description vector, anchor box position information and image feature vector, perform semantic similarity optimization query in the multimodal feature sequence to iteratively update the feature description vector and anchor box position information;

[0077] Step S303: When the iterative update meets the preset conditions, a target anchor box is generated in the target visual image based on the anchor box position information after the iterative update.

[0078] According to some embodiments provided in this application, in the process of generating target anchor boxes for recognizing target objects in a target visual image based on feature description vectors and image feature vectors, the anchor box position information can be obtained first based on the feature description vectors and image feature vectors. Then, based on the feature description vectors, anchor box position information, and image feature vectors, semantic similarity optimization queries are performed in a multimodal feature sequence to iteratively update the feature description vectors and anchor box position information. When the iterative update meets preset conditions, the target anchor box is generated in the target visual image based on the iteratively updated anchor box position information. Specifically, after obtaining the feature description vectors and image feature vectors, the target visual image can be reconstructed according to each image feature vector, and the anchor box position information can be obtained by locating the target visual image according to the feature description vectors. In some exemplary embodiments of this application, after obtaining the anchor frame position information, semantic similarity optimization queries can be further performed in a multimodal feature sequence based on the feature description vector, anchor frame position information, and image feature vector to iteratively update the feature description vector and anchor frame position information. It should be understood that iteratively performing semantic similarity optimization queries in a multimodal feature sequence based on the feature description vector, anchor frame position information, and image feature vector to optimize and update the anchor frame position information enables the final generated target anchor frame to more accurately match the actual position of the target object in the target visual image. It should be noted that when the iterative update meets preset conditions, a target anchor frame can be generated in the target visual image based on the iteratively updated anchor frame position information. The preset conditions can be set in various ways, including, but not limited to, setting the preset conditions to the number of iterations reaching a preset threshold, setting the preset conditions to the adjustment value of the anchor frame relative to the previous round being less than a preset threshold, etc.

[0079] According to some exemplary embodiments of this application, based on feature description vectors, anchor box position information, and image feature vectors, semantic similarity optimization queries are performed in a multimodal feature sequence to iteratively update the feature description vectors and anchor box position information. This includes: in each iteration, performing semantic similarity optimization queries in the multimodal feature sequence based on the feature description vectors to update the feature description vectors; and updating the anchor box position information based on the image feature vectors and the updated feature description vectors. It should be noted that as the iterations continue, the final generated target anchor box can increasingly accurately match the actual position of the target object in the target visual image.

[0080] In some specific embodiments of this application, the feature description vector and anchor box position information can be iteratively updated in the anchor box decoder. The anchor box decoder can be composed of six stacked Transformer decoder layers. Each decoder layer includes a self-attention sublayer, a cross-attention sublayer, and a feedforward sublayer. The self-attention sublayer performs similarity calculations, the cross-attention sublayer queries the image feature vector as a key-value pair, and the feedforward sublayer enhances the model's expressive power through dimensionality adjustment. It should be noted that the query request of each Transformer decoder layer in the anchor box decoder can be composed of a feature description vector and anchor box position information. The feature description vector and anchor box position information of the first Transformer layer are the outputs of the prototype decoder. The feature description vectors of the remaining layers are updated to the outputs of the previous Transformer layer, and the anchor box position information is updated according to the updated feature description vector. In some exemplary embodiments of this application, the method for updating the anchor frame position information can be to update the center point coordinates, width and height information in the anchor frame position information according to the feature description vector after each update, combined with the image feature vector. Finally, when the iterative update meets the preset conditions, the target anchor frame is generated in the target visual image by dimensionality reduction based on the iteratively updated anchor frame position information. The anchor frame can contain four dimensions, for example, x and y represent the center point of the anchor frame, and w and h represent the width and height of the anchor frame.

[0081] Reference Figure 4 According to some embodiments of this application, the anchor box position information is updated based on the image feature vector and the updated feature description vector, including:

[0082] Step S401: Based on the image feature vector and the updated feature description vector, obtain the optimized anchor box information;

[0083] Step S402: Compare the optimized anchor box information with the feature description vector to obtain the predicted offset;

[0084] Step S403: Update the anchor frame position information based on the predicted offset.

[0085] According to some preferred embodiments of this application, in the process of updating anchor box position information based on image feature vectors and updated feature description vectors, optimized anchor box information can first be obtained based on the image feature vectors and updated feature description vectors. The optimized anchor box information is then compared with the feature description vectors to obtain a predicted offset. The anchor box position information is then updated based on the predicted offset. It should be noted that the optimized anchor box information is anchor box information that more closely matches the actual position of the target object in the target visual image, obtained based on the image feature vectors and updated feature description vectors. Based on the anchor box position information before the update and the optimized anchor box information, the predicted offset can be obtained by comparison, thereby completing the update of the anchor box position information based on the predicted offset.

[0086] Reference Figure 5 According to some embodiments of this application, forming an image feature vector based on a target visual image and forming a text feature vector based on pointer expression information includes:

[0087] Step S501: Perform image patch encoding on the target visual image to obtain the image feature vector;

[0088] Step S502: Extract semantic features from the indicated expression information to obtain the text feature vector.

[0089] According to some embodiments provided in this application, after acquiring the target visual image, it is necessary to segment the target visual image and extract image features through an image feature network to obtain an image feature vector. In some more specific embodiments, the image feature network can be a hierarchical model containing 12 layers of Swin Transformer sub-modules. Each layer of Swin Transformer sub-module is used to reduce the resolution of the input feature map and expand the receptive field layer by layer, thereby obtaining an image feature vector of a fixed size. According to some embodiments provided in this application, after acquiring the index expression information, it is necessary to perform text segmentation and text feature extraction on the index expression information through a natural language network to form a text feature vector. It should be noted that the text feature vector reflects the text semantic information in the index expression information. In some more specific embodiments of this application, the natural language network can be a trained semantic recognition model, wherein the natural language network can include 12 layers of BERT Transformer sub-modules. After extracting semantic features from the index expression information layer by layer, a feature vector of fixed length for each word is obtained.

[0090] Reference Figure 6 According to some embodiments of this application, correlation mapping processing is performed on image feature vectors and text feature vectors to generate multimodal feature sequences, including:

[0091] Step S601: Construct a multimodal encoder based on a self-attention mechanism. The multimodal encoder includes a multi-head self-attention layer.

[0092] Step S602: Through a multi-head self-attention layer, feature association information is established based on image feature vectors and text feature vectors;

[0093] Step S603: Based on the feature association information, the image feature vector and the text feature vector are concatenated into a multimodal feature sequence.

[0094] It should be understood that after obtaining the image feature vector and the text feature vector, the two unimodal feature sequences can be concatenated to generate a multimodal feature sequence. It should be noted that in some embodiments of this application, during the process of generating a multimodal feature sequence by performing correlation mapping processing on the image feature vector and the text feature vector, it is necessary to first establish feature association information between the image feature vector and the text feature vector, and then concatenate the image feature vector and the text feature vector based on the feature association information to obtain the multimodal feature sequence. It should be understood that the feature association information reflects the semantic correlation between the image feature vector and the text feature vector, that is, the autocorrelation of the multimodal sequence. In some specific embodiments of this application, a multimodal encoder can be pre-constructed based on a self-attention mechanism. Specifically, the multimodal encoder can consist of six Transformer sub-modules, each containing a multi-head self-attention layer and a forward propagation network layer. The self-attention layer calculates the autocorrelation of multimodal sequences in multiple subspaces to form feature association information, facilitating the concatenation of image feature vectors and text feature vectors to ultimately generate a multimodal feature sequence. The forward propagation layer can use a rectified linear unit (ReLU) as the activation function for neurons, consisting of two layers of linear transformation and a ReLU nonlinear transformation, further enhancing the expressive power of the multimodal encoder. Specifically, enhancing the expressive power of the multimodal encoder means that the forward propagation layer can perform dimensionality upscaling or dimensionality reduction on single-modal feature sequences such as image feature vectors and text feature vectors, or on multimodal sequences, to give the multimodal feature sequences more expressive space. According to some exemplary embodiments of this application, multimodal feature sequences can be encoded by mapping to a two-dimensional sinusoidal embedding space. For image feature vectors, a two-dimensional sinusoidal encoder can be used to encode the position of each image block, mapping the image feature vectors into a two-dimensional sinusoidal embedding space. For text feature vector encoding, a multilayer perceptron can be used to convert the text feature vectors into two-dimensional text coordinates using the semantic information of the text words and the position of the text words in the sequence as two dimensions. Then, a two-dimensional sinusoidal encoder is used to encode the two-dimensional text coordinates and map them into a two-dimensional sinusoidal embedding space. Finally, the semantic features and positions of the multimodal sequence are embedded together into the multimodal encoder for multimodal information fusion, resulting in a multimodal feature sequence mapped into a two-dimensional sinusoidal embedding space.

[0095] Reference Figure 7 The target recognition model 700 according to a second aspect embodiment of this application includes:

[0096] The data acquisition module 701 is used to acquire the target visual image and the pointer expression information, and the pointer expression information is used to describe the target object in the target visual image;

[0097] The multimodal encoder 702 is used to form an image feature vector based on the target visual image, and to form a text feature vector based on the index expression information, and to perform correlation mapping processing on the image feature vector and the text feature vector to generate a multimodal feature sequence.

[0098] The prototype decoder 703 is used to perform an initial semantic similarity query in a multimodal feature sequence based on text feature vectors to obtain a feature description vector corresponding to the target object.

[0099] Anchor frame decoder 704 is used to generate target anchor frames in a target visual image for identifying target objects based on feature description vectors and image feature vectors.

[0100] Figure 8 An electronic device 800 according to an embodiment of this application is shown. The electronic device 800 includes a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the computer program is executed, it is used to perform the target recognition method described above.

[0101] The processor 801 and the memory 802 can be connected via a bus or other means.

[0102] The memory 802, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the target recognition method described in the embodiments of this application. The processor 801 implements the above-described target recognition method by running the non-transitory software program and instructions stored in the memory 802.

[0103] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function. The data storage area may store the target identification method described above. Furthermore, the memory 802 may include high-speed random access memory 802, and may also include non-transitory memory 802, such as at least one storage device, flash memory, or other non-transitory solid-state storage device. In some embodiments, the memory 802 may optionally include memory 802 remotely located relative to the processor 801, and these remote memories 802 may be connected to the electronic device 800 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0104] The non-transient software program and instructions required to implement the target recognition method described above are stored in memory 802. When executed by one or more processors 801, the target recognition method described above is executed, for example, executing... Figure 1 Method steps S101 to S105, Figure 2Method steps S201 to S202, Figure 3 Method steps S301 to S303, Figure 4 Method steps S401 to S403, Figure 5 Method steps S501 to S502, Figure 6 The method steps S601 to S603 are described in the text.

[0105] This application also provides a computer-readable storage medium storing computer-executable instructions for performing the target recognition method described above.

[0106] In one embodiment, the computer-readable storage medium stores computer-executable instructions that are executed by one or more control processors, for example, executing... Figure 1 Method steps S101 to S105, Figure 2 Method steps S201 to S202, Figure 3 Method steps S301 to S303, Figure 4 Method steps S401 to S403, Figure 5 Method steps S501 to S502, Figure 6 The method steps S601 to S603 are described in the text.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, storage device storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is well known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. It should also be understood that the various implementation methods provided in this application can be arbitrarily combined to achieve different technical effects.

[0109] The above provides a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A target recognition method characterized by, include: Acquire a target visual image and pointer expression information, wherein the pointer expression information is used to describe the target object in the target visual image; An image feature vector is formed based on the target visual image, and a text feature vector is formed based on the index expression information; The image feature vector and the text feature vector are subjected to correlation mapping processing to generate a multimodal feature sequence; A prototype decoder is constructed based on a cross-attention mechanism, which includes a cross-attention layer and a weighted average pooling layer. The text feature vector is used as the initial query request, and the semantic similarity initial query is performed on the multimodal feature sequence through the cross-attention layer to obtain the feature description vector; Based on the feature description vector and the image feature vector, a target anchor box for identifying the target object is generated in the target visual image; The step of generating a target anchor box for identifying the target object in the target visual image based on the feature description vector and the image feature vector includes: The weighted average pooling layer performs dimensionality reduction on the feature description vector and the image feature vector to obtain a target visual image, and the weighted average pooling layer generates the target anchor box for identifying the target object in the target visual image based on the feature description vector.

2. The method of claim 1, wherein, The step of generating a target anchor box for identifying the target object in the target visual image based on the feature description vector and the image feature vector includes: Based on the feature description vector and the image feature vector, the anchor frame position information is obtained; Based on the feature description vector, the anchor box position information, and the image feature vector, a semantic similarity optimization query is performed in the multimodal feature sequence to iteratively update the feature description vector and the anchor box position information; When the iterative update meets the preset conditions, the target anchor frame is generated in the target visual image based on the anchor frame position information after the iterative update.

3. The method of claim 2, wherein, The step of performing semantic similarity optimization query on the multimodal feature sequence based on the feature description vector, the anchor box position information, and the image feature vector, to iteratively update the feature description vector and the anchor box position information, includes: In each round of the iterative update, the semantic similarity optimization query is performed on the multimodal feature sequence based on the feature description vector to update the feature description vector, and the anchor box position information is updated based on the image feature vector and the updated feature description vector.

4. The method of claim 3, wherein, The step of updating the anchor box position information based on the image feature vector and the updated feature description vector includes: Based on the image feature vector and the updated feature description vector, the optimized anchor box information is obtained; The optimized anchor frame information is compared with the feature description vector to obtain the predicted offset; The anchor frame position information is updated based on the predicted offset.

5. The method according to any one of claims 1 to 4, characterized in that, The step of forming an image feature vector based on the target visual image and forming a text feature vector based on the index expression information includes: The target visual image is subjected to image patch encoding to obtain the image feature vector; Semantic features are extracted from the indicated information to obtain the text feature vector.

6. The method of claim 5, wherein, The step of performing correlation mapping processing on the image feature vector and the text feature vector to generate a multimodal feature sequence includes: A multimodal encoder is constructed based on a self-attention mechanism, and the multimodal encoder includes a multi-head self-attention layer; Feature association information is established based on the image feature vector and the text feature vector through the multi-head self-attention layer; Based on the feature association information, the image feature vector and the text feature vector are concatenated to form the multimodal feature sequence.

7. A target recognition model, characterized in that, include: The data acquisition module is used to acquire target visual images and pointer expression information, wherein the pointer expression information is used to describe the target object in the target visual image; A multimodal encoder is used to form an image feature vector based on the target visual image, and to form a text feature vector based on the index expression information, and to perform correlation mapping processing on the image feature vector and the text feature vector to generate a multimodal feature sequence; The prototype decoder includes a cross-attention layer and a weighted average pooling layer. The cross-attention layer is used to perform the initial semantic similarity query on the multimodal feature sequence based on the text feature vector to obtain a feature description vector. An anchor box decoder is used to generate target anchor boxes for recognizing the target object in the target visual image based on the feature description vector and the image feature vector. The step of generating a target anchor box for identifying the target object in the target visual image based on the feature description vector and the image feature vector includes: The weighted average pooling layer performs dimensionality reduction on the feature description vector and the image feature vector to obtain a target visual image, which is then used to generate the target anchor box for identifying the target object in the target visual image based on the feature description vector in the weighted average pooling layer.

8. An electronic device, comprising: include: The device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the target recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the target recognition method as described in any one of claims 1 to 6.

Images