Target identification method and device, electronic equipment and product

Abstract

The invention provides a target recognition method and device, electronic equipment and a product, and is applied to the technical field of target recognition. The method comprises the steps of obtaining first text description information of a to-be-recognized object and a plurality of to-be-detected images, wherein the first text description information comprises descriptions of a plurality of components of the to-be-recognized object; the first text description information and the multiple to-be-detected images are input into a recognition model, a target image output by the recognition model is obtained, the target image is a to-be-detected image containing the to-be-recognized object, and the to-be-recognized object is a to-be-detected image containing the to-be-recognized object. The recognition model has a function of aligning local image features and local text features of the to-be-recognized object.

Description

Technical Field

[0001] This application relates to the field of target recognition technology, and in particular to a target recognition method, device, electronic device and product. Background Technology

[0002] With the development of large model technology, it is possible to identify objects such as people and things through models.

[0003] In the exemplary technology, textual description information of the identified object and multiple surveillance images are input into the model, so that the model can retrieve surveillance images containing the identified object from among the multiple surveillance images.

[0004] However, the differences between text and images make it difficult for the model to match text with images, making it difficult for the model to retrieve images containing the object to be identified, that is, the recognition accuracy of images containing the object to be identified is low. Summary of the Invention

[0005] To address the aforementioned problems, this application proposes a target recognition method, apparatus, electronic device, and product to solve the problem of low accuracy in identifying the target.

[0006] The first aspect of this application provides a target recognition method, including:

[0007] Acquire first textual description information of the object to be identified and multiple images to be detected, wherein the first textual description information contains descriptions of multiple parts of the object to be identified;

[0008] The first text description information and multiple images to be detected are input into the recognition model to obtain the target image output by the recognition model. The target image is an image to be detected containing the object to be identified. The recognition model has the function of aligning the local image features and local text features of the object to be identified.

[0009] In some embodiments, before obtaining the first textual description information of the object to be identified and multiple images to be detected, the method further includes:

[0010] Multiple training samples are obtained, wherein the training samples include region images of multiple components of the sample object in the training image corresponding to the sample object, second textual description information of the training image, and third textual description information corresponding to each region image;

[0011] The preset model is trained based on each of the training samples to obtain the recognition model, wherein the preset model learns the alignment of local image features and local text features based on each of the region images, each of the third text description information and the second text description information in the training samples.

[0012] In some embodiments, the preset model includes an image feature encoder, a text feature encoder, and a local feature alignment module, wherein,

[0013] The image feature encoder is used to encode features of each of the region images to obtain multiple local image feature vectors;

[0014] The text feature encoder is used to perform feature encoding on the second text description information to obtain a first text feature vector, and to perform feature encoding on each of the third text description information to obtain each of the second text feature vectors.

[0015] The local feature alignment module is used to perform alignment learning of local image features and local text features based on each of the local image feature vectors, the first text feature vector, and each of the second text feature vectors.

[0016] In some embodiments, the training samples further include a mask image corresponding to each of the region images and a mask text corresponding to each of the third text description information, and the preset model further includes a global feature alignment module, wherein...

[0017] The image feature encoder is further configured to perform feature encoding on each of the masked images to obtain multiple masked image feature vectors, and to perform feature encoding on the training images to obtain a global image feature vector.

[0018] The text feature encoder is also used to perform feature encoding on each of the masked texts to obtain multiple masked text feature vectors;

[0019] The global feature alignment module is used to perform alignment learning from local features to global features based on the global image feature vector, each of the mask image feature vectors, each of the mask texts, and the first text feature vector. The learning results of the global feature alignment module and the learning results of the local feature alignment module are used to update the network parameters of the preset model.

[0020] In some embodiments, the global feature alignment module includes a mask image submodule and a mask text submodule, wherein,

[0021] The mask image submodule is used to recover the masked area in the mask image corresponding to the feature vector of the mask image;

[0022] The masked text submodule is used to recover the masked text that is hidden in the masked text corresponding to the masked text feature vector.

[0023] In some embodiments, the local feature alignment module performs alignment learning of local image features and local text features based on each of the local image feature vectors, the first text feature vector, and each of the second text feature vectors, including:

[0024] The first text feature vector and each of the second text feature vectors are concatenated to obtain a concatenated feature vector.

[0025] The concatenated feature vector is subjected to feature extraction to obtain multiple sub-feature vectors;

[0026] Alignment learning between local image features and local text features is performed based on each of the sub-feature vectors and each of the local feature vectors.

[0027] In some embodiments, the training samples further include a mask image corresponding to each of the region images and a mask text corresponding to each of the third text description information. The step of obtaining multiple training samples includes:

[0028] The training image of the sample object is input into the text image fusion model to obtain the second text description information, each of the third text description information, the mask image corresponding to the region image of the multiple parts of the sample object in the training image, and the mask text corresponding to each of the third text description information output by the text image fusion model.

[0029] Training samples corresponding to the training images are constructed based on the training images, the second text description information, each of the third text description information, each of the mask images, and each of the mask texts.

[0030] A second aspect of this application discloses a target recognition device, comprising:

[0031] The acquisition module is used to acquire first textual description information of the object to be identified and multiple images to be detected. The first textual description information includes descriptions of multiple parts of the object to be identified.

[0032] The input module is used to input the first text description information and multiple images to be detected into the recognition model to obtain the target image output by the recognition model. The target image is an image to be detected containing the object to be identified. The recognition model has the alignment function of the local image features and local text features of the object to be identified.

[0033] A third aspect of this application provides an electronic device, including a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the target recognition method as described in the first aspect or any embodiment of the first aspect by running the program in the memory.

[0034] The fourth aspect of this application provides a chip including a processor and a data interface, wherein the processor reads and runs a program stored in a memory through the data interface to perform a target recognition method as described in the first aspect or any embodiment of the first aspect.

[0035] The fifth aspect of this application provides a storage medium storing a computer program, which, when executed by a processor, implements the target recognition method as described in the first aspect or any embodiment of the first aspect.

[0036] The sixth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the target recognition method as described in the first aspect or any embodiment of the first aspect.

[0037] This application proposes a target recognition method, apparatus, electronic device, and product. It acquires textual description information of the object to be recognized and multiple images to be detected. The textual description information and multiple images to be detected are input into a recognition model to obtain a target image containing the object to be recognized, output by the recognition model. Because the recognition model has the function of aligning local image features and local text features of the object to be recognized, it can align the textual description of the parts of the object to be recognized in the textual description information with the local images in the images to be detected. This allows the recognition model to accurately match text and images, improving the accuracy of object recognition. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the implementation environment involved in the embodiments of this application;

[0040] Figure 2 This is a flowchart illustrating the target identification method provided according to the embodiments of this application. Figure 1 ;

[0041] Figure 3 This is a flowchart illustrating the target identification method provided according to the embodiments of this application. Figure 2 ;

[0042] Figure 4 This is a flowchart illustrating the workflow of the local feature alignment module provided in an embodiment of this application.

[0043] Figure 5 This is a flowchart illustrating the workflow of the global feature alignment module provided in the embodiments of this application.

[0044] Figure 6 This is a flowchart illustrating the target identification method provided according to the embodiments of this application. Figure 3 ;

[0045] Figure 7 This is a simplified training diagram of the recognition model provided according to an embodiment of this application;

[0046] Figure 8 This is a schematic diagram of the target recognition device provided according to an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] With the development of large model technology, it is possible to identify objects such as people and things through models.

[0050] In the exemplary technology, textual description information of the identified object and multiple surveillance images are input into the model, so that the model can retrieve surveillance images containing the identified object from among the multiple surveillance images.

[0051] However, the differences between text and images make it difficult for the model to match text with images, making it difficult for the model to retrieve images containing the object to be identified, that is, the recognition accuracy of images containing the object to be identified is low.

[0052] To address the aforementioned problems, embodiments of this application provide a target recognition method, apparatus, electronic device, and product, applicable to the field of target recognition technology. In this target recognition method, textual description information of the object to be recognized and multiple images to be detected are acquired. The textual description information and multiple images to be detected are input into a recognition model, resulting in a target image containing the object to be recognized output by the recognition model. Because the recognition model has the function of aligning local image features and local text features of the object to be recognized, the recognition model can align the textual description of the components of the object to be recognized in the textual description information with the local images in the images to be detected. This allows the recognition model to accurately match text and images, improving the accuracy of object recognition.

[0053] Exemplary Implementation Environment

[0054] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the implementation environment according to the embodiments of this application. The implementation environment of this application includes a target recognition device 100, which is equipped with a recognition model 110. The target recognition device 100 acquires textual description information of the object to be recognized and multiple images to be detected. The textual description information contains descriptions of multiple components of the object to be recognized. The target recognition device 100 inputs the textual description information and the multiple images to be detected into the recognition model 110. Because the recognition model 110 has the function of aligning local image features and local text features of the object to be recognized, the recognition model 110 can output a target image containing the object to be recognized; that is, the target recognition device 100 outputs a target image.

[0055] Exemplary methods

[0056] Please see Figure 2 In one exemplary embodiment, a target recognition method is provided, which includes the following steps:

[0057] Step S201: Obtain first text description information of the object to be identified and multiple images to be detected. The first text description information contains descriptions of multiple parts of the object to be identified.

[0058] In this embodiment, the executing entity is a target recognition device. For ease of description, the term "device" will be used to refer to the target recognition device below. The device can be a server, a computer, any terminal device with target recognition function, or a monitoring backend physical device or a backend virtual device.

[0059] The device acquires textual description information of the object to be identified. The object to be identified can be a tracked person or a monitored object, such as a vehicle to be monitored. The textual description information is defined as first textual description information, which contains descriptions of multiple components of the object to be identified. For example, if the object is a vehicle, the first textual description information may include descriptions of multiple components such as headlights, doors, roof, windows, wheels, and body. For instance, the body description might include the color of the vehicle body. The first textual description information can also be text information entered by the user.

[0060] The database stores videos captured by monitoring equipment. These videos are processed into images, all defined as images to be detected. Each image may contain an object to be identified. For example, the device can obtain time information input by the user, extract videos captured at that time from the database, and process the videos to obtain multiple images to be detected. For instance, if a user needs to query for red vehicles with no windows and high headlights that appeared between 9:00 am and 10:00 am today, the device will extract videos captured between 9:00 am and 10:00 am today from the database, process them to obtain multiple images to be detected, and obtain a first text description based on the user's description of the red vehicles with no windows and high headlights.

[0061] Step S202: Input the first text description information and multiple images to be detected into the recognition model to obtain the target image output by the recognition model. The target image is the image to be detected containing the object to be identified. The recognition model has the alignment function of local image features and local text features of the object to be identified.

[0062] After obtaining the first text description information and multiple images to be detected, the first text description information and multiple images to be detected are input into the recognition model, and the recognition model can output the target image, which is the image to be detected containing the object to be identified.

[0063] For example, the recognition model has the function of aligning local image features and local text features of the object to be recognized. Therefore, the model can find the component image features that match the text description of each component in the first text description information in the image to be detected, and calculate the similarity between the local text features corresponding to the text description and the compared component image features to obtain the similarity of the image to be detected. Based on the similarity of the image to be detected, the target image is determined among the images to be detected. For example, the image to be detected with a similarity greater than a preset threshold can be used as the target image. If there are multiple images to be detected with a similarity greater than the preset threshold, these images can all be used as intermediate images, and the user who entered the first text description information can be prompted to select the target image from the multiple intermediate images.

[0064] It should be noted that the target recognition method provided in this embodiment is applicable to scenarios including but not limited to cross-modal vehicle re-identification and cross-modal pedestrian re-identification. The object to be identified can be a vehicle to be searched in a traffic monitoring scenario or a person to be searched in a monitoring scenario. The specific application scenarios mentioned above can be road monitoring scenarios or other types of monitoring scenarios, that is, the device can be a device in a road monitoring system or a device in other types of monitoring systems.

[0065] In this embodiment, textual description information of the object to be identified and multiple images to be detected are obtained. The textual description information and multiple images to be detected are input into the recognition model to obtain the target image containing the object to be identified, which is output by the recognition model. Since the recognition model has the function of aligning local image features and local text features of the object to be identified, the recognition model can align the textual description of the parts of the object to be identified in the textual description information with the local images in the images to be detected. This enables the recognition model to accurately match the text and the image, thereby improving the recognition accuracy of the object.

[0066] Please see Figure 3 In yet another exemplary embodiment, a target recognition method is provided, based on Figure 2 In the embodiment shown, prior to step S201, the method further includes:

[0067] Step S301: Obtain multiple training samples. The training samples include region images of multiple parts of the sample object in the training image corresponding to the sample object, second text description information of the training image, and third text description information corresponding to each region image.

[0068] In this embodiment, the device can train a recognition model. For example, the device acquires multiple training samples, which include multiple components of a sample object and region images in the training image corresponding to the sample object.

[0069] Training images refer to images containing sample objects. Region images can exist in the form of bounding boxes in training images. For example, a training image may contain 6 detection boxes, each bounding box being a region image of a part of a sample object in the training image. For instance, a vehicle in a training image may have bounding boxes containing wheels, headlights, windows, roof, doors, and body, etc.

[0070] The training samples also include second textual descriptions of the training images, which refer to the overall description of the sample objects in the training images. The training samples also include third textual descriptions of each region image, namely, textual descriptions of the wheels, headlights, windows, roof, doors, and body.

[0071] Step S302: Train the preset model according to each training sample to obtain the recognition model. The preset model learns the alignment of local image features and local text features based on each region image, each third text description information and the second text description information in the training samples.

[0072] After obtaining the training samples, the preset model can be trained using these samples to obtain the recognition model. The preset model learns the alignment of local image features and local text features based on the regional images, third-party text descriptions, and second-party text descriptions in the training samples. The preset model can be a deep learning model, which has a learning task set up to learn the alignment of local image features and local text features.

[0073] In this embodiment, the device acquires various training samples, thereby training the preset model through various training versions, thus obtaining a model with the function of aligning local image features with local text features.

[0074] In one embodiment, the preset model includes an image feature encoder, a text feature encoder, and a local feature alignment module.

[0075] An image feature encoder is used to encode features of various regions of an image to obtain multiple local image feature vectors. For example, an image feature encoder can include multiple self-attention layers and feedforward layers. For instance, there can be 12 self-attention layers. That is, by inputting each region image into the image feature encoder composed of 12 self-attention layers and feedforward layers, the encoded local image feature vector for each region image can be obtained.

[0076] A text feature encoder can be an encoder composed of multiple self-attention layers and feedforward layers. By inputting the second text description information into the text feature encoder, a first text feature vector can be obtained. Similarly, by inputting each third text description information into the text feature encoder, each second text feature vector can be obtained. In essence, the text feature encoder is used to encode features from the second and third text description information.

[0077] The local feature alignment module is a component-level local fine-grained alignment module. It is used to learn the alignment of local image features and local text features based on each local image feature vector, the first text feature vector, and each second text feature vector.

[0078] In one example, the training samples are labeled with labels indicating the relationship between the third text description information and the region image. For example, the headlight description information corresponds to the region image containing the headlights. This correspondence serves as the label. Therefore, the local feature alignment module learns the alignment of local image features and local text features by studying the feature relationship between the second text feature vector in the first text feature vector and the second text feature vector that has a corresponding relationship with the second text feature vector.

[0079] In another example, the local feature alignment module concatenates the first text feature vector and each of the second text feature vectors to obtain a concatenated feature vector. It then extracts features from this concatenated feature vector to obtain multiple sub-feature vectors. Based on these sub-feature vectors and the local feature vectors, it performs alignment learning between local image features and local text features. Specifically, refer to... Figure 4 The process of the local feature alignment module for learning the alignment of local image features and local text features is as follows:

[0080] 1. The input image is encoded by an image feature encoder to obtain an image feature vector. The input image can be various regions of the training image, and the image feature vector is the local image feature vector.

[0081] 2. The global text (second text description information) and component text (third text description information) are feature encoded by a text feature encoder to obtain a first text feature vector and multiple second text feature vectors;

[0082] 3. The first text feature vector and the second text feature vector are concatenated by the controller C to obtain a concatenated feature vector, which is used as the input of the key and value in the cross attention of the component decoder in the local feature alignment module;

[0083] 4. The component decoder is equipped with an adaptive learning component query token. The component query token is the learning parameter corresponding to the adaptive learning in the component decoder. Adaptive text feature extraction is performed from the concatenated feature vector through the component query token.

[0084] 5. Each global image feature vector is input to the component decoder. The component decoder uses the extracted text feature vector and global image feature vector to perform alignment learning between local text features and local image features. The learning result is used to calculate the image text matching loss parameter. The image text matching loss parameter is then used to promote the learning of the component query token. That is, the learning parameters are updated by the image text matching loss function.

[0085] In this embodiment, the local feature alignment module utilizes local image-text pairs to enable the trained preset model to perceive and align the corresponding local text features in the global text features with the corresponding local image features in the global visual features, thereby enabling the recognition model to perceive component-level image-text alignment information.

[0086] In one embodiment, the training samples, in addition to including images of each region, second textual description information of the training images, and third textual description information corresponding to each region image, also include a mask image corresponding to each region image and a mask text corresponding to each third textual description information. The preset model, in addition to including an image feature encoder, a text feature encoder, and a local feature alignment module, also includes a global feature alignment module.

[0087] The global feature alignment module further aligns local features to global features, building upon the local information alignment of the local feature alignment module.

[0088] The image feature encoder is also used to encode features for each mask image to obtain multiple mask image feature vectors, and to encode features for the training image to obtain a global image feature vector.

[0089] The text feature encoder is also used to perform feature encoding on each masked text to obtain multiple masked text feature vectors;

[0090] The global feature alignment module is used to perform alignment learning from local features to global features based on the global image feature vector, the feature vectors of each mask image, each mask text, and the first text feature vector. The learning results of the global feature alignment module and the learning results of the local feature alignment module are used to update the network parameters of the preset model.

[0091] Specifically, the mask image comprises the masked region and the unmasked region. The third text description information contains a text segment containing multiple words. Randomly masking the words in the text yields the mask text corresponding to the third text description information. In one example, the global feature alignment module recovers the masked portion in the mask image using the global image feature vector, thereby learning the alignment from local image features to global image features; the global feature alignment module recovers the masked words in the mask text using the first text feature vector, thereby learning the alignment from local text features to global text features.

[0092] In another example, the global feature alignment module includes a mask image submodule and a mask text submodule. The mask image submodule is used to recover the masked areas in the mask image corresponding to the mask image feature vector; the mask text submodule is used to recover the masked text in the mask text corresponding to the mask text feature vector. Specifically, refer to... Figure 5 The global feature alignment module's alignment learning process from local features to global features is as follows:

[0093] 1. Input the mask image feature vector as the query (Q), the first text feature vector as the key (K) and the value (V) into the mask image submodule;

[0094] 2. The mask image submodule includes a cross-autonomous layer, a self-attention layer, a feedforward layer, and a mask image modeling prediction head. It uses four layers of cross-autonomous layer, self-attention layer, and feedforward layer to process the mask image feature vector and the first text feature vector. The first text feature vector is used as a key and value in the cross-attention calculation.

[0095] 3. Use the masked image modeling prediction head (referred to as image prediction head) to recover the masked image part in the feature vector of the masked image, and calculate the pixel-level point-to-point distance with the original input image features as the optimization loss;

[0096] 4. Input the masked text feature vector as query Q, the global image feature vector as key K and value V into the masked text sub-model;

[0097] 5. The masked text sub-model includes a cross-attention layer, a self-attention layer, a feedforward layer, and a masked text modeling prediction head. It utilizes one cross-attention layer, four self-attention layers, and a feedforward layer to process the masked text feature vector and the global image feature vector. The global image feature vector is used as a key and value in the cross-attention calculation.

[0098] 6. Using the masked text modeling prediction head (referred to as the text prediction head), the masked text portion in the masked text feature vector is recovered. The recovered text is, for example, "yellow", "rectangle", and "skylight". The text prediction head calculates the point-to-point distance based on the first text feature vector as the optimization loss. That is, the optimization loss calculated by the masked image modeling prediction head and the optimization loss calculated by the masked text modeling prediction head are used to update the parameters of the global feature alignment module.

[0099] In this embodiment, a global feature alignment module randomly masks words in sentences within the text, and uses bounding boxes to mask vehicles in the image, before restoring the image. Another module is used to restore the masked portions of both the image and text, thus achieving local-to-global alignment between the image and text modules.

[0100] Please see Figure 6 In another exemplary embodiment, a target recognition method is provided, based on Figure 3 In the embodiment shown, step S301 includes:

[0101] Step S601: Input the training image of the sample object into the text image fusion model to obtain the second text description information, each third text description information, the mask image corresponding to the region image of multiple parts of the sample object in the training image, and the mask text corresponding to each third text description information output by the text image fusion model.

[0102] In this embodiment, the device acquires training images of sample objects and inputs these training images into a text-image fusion model. The text-image fusion model improves the performance of visual reasoning and object detection tasks. It is obtained by adding a detector from a deep learning model with a self-attention mechanism to a pre-trained model based on language and images.

[0103] The text-image fusion model is based on training images and outputs second text description information, images of each region, third text description information corresponding to the region images, mask images corresponding to the region images, and mask text corresponding to the third text description information.

[0104] Step S602: Construct training samples corresponding to the training images based on the training images, the second text description information, each third text description information, each mask image, and each mask text.

[0105] Once the second text description information, each region image, the third text description information corresponding to the region image, the mask image corresponding to the region image, and the mask text corresponding to the third text description information are obtained, training samples corresponding to the training image can be constructed based on the training image, the second text description information, each region image, the third text description information corresponding to the region image, the mask image corresponding to the region image, and the mask text corresponding to the third text description information.

[0106] In this embodiment, the text-image fusion model can obtain the detection boxes corresponding to component words in the sample object through text prediction detection boxes, such as headlights, rearview mirrors, headlights, and roof. The text-image fusion model can obtain the image-text pair information at the component level of the sample object, realizing refined acquisition of image-text pair information and improving the training effect of the model. On this basis, the local feature alignment module and the global feature alignment module effectively fuse global alignment and local alignment.

[0107] Based on the above embodiments, referring to Figure 7 The process of training the recognition model in this application is briefly described below, with specific steps as follows:

[0108] 1. Obtain local fine-grained image-text pairs. This is mainly achieved through a text-image fusion model. Specifically, by providing the text-image fusion model with training images and the text of each component of the object to be identified in the training images (e.g., six component texts), multiple component detection boxes are obtained. This forms all the information in the training samples: the complete image, the six component description texts, the component mask images of the six sets of component detection boxes masking the complete image, the description text of the overall image, and the mask text of random word masks.

[0109] 2. Input the complete image and the masked image into an image feature encoder consisting of 12 layers of self-attention and feedforward layers for feature encoding to obtain the feature vector of the masked image and the feature vector of the global image;

[0110] 3. Input the six component description texts (descriptions of each component of the vehicle), the overall image description text (complete description of the vehicle), and the mask text of the random word mask (mask text information) into a text feature encoder composed of 12 layers of self-attention and feedforward layers for feature encoding to obtain component text feature vectors (second text feature vectors), complete text feature vectors (first text feature vectors), and mask text feature vectors.

[0111] 4. Input the global image feature vector, component text feature vector, and complete text feature vector into the local feature alignment module for local alignment learning;

[0112] 5. Input the global image feature vector, masked image feature vector, complete text feature vector, and masked text feature vector into the global feature alignment module to perform local-to-global alignment learning;

[0113] 6. Calculate the identity loss parameter IDloss and the regression conformance distance loss parameter SDM on the global image feature vector and the complete text feature vector to perform global image and text alignment learning.

[0114] This application evaluates its target recognition method by comparing its performance with state-of-the-art cross-modal re-identification methods on a set dataset, in order to verify the excellent performance of the recognition model trained above, as detailed in Table 1.

[0115] This application selects Rank-1, Rank-5, Rank-10, and mAP—the most commonly used evaluation metrics for cross-modal re-identification—in the dataset for quantitative evaluation of the method. Rank-1, Rank-5, and Rank-10 refer to whether the correct match is found in the top 1, top 5, and top 10 images of the similarity scores obtained during the inference process between the current text and all images. If the correct match is found, it indicates a successful match. mAP is a comprehensive metric that considers both precision and recall of the search results. mAP is obtained by calculating the average precision for each query and then averaging it across all queries. The results show that the method proposed in this application can be applied to object tracking datasets and exhibits significant improvements compared to other methods.

[0116] Table 1

[0117]

[0118] This application introduces a local fine-grained image-text pair (local feature alignment module) to integrate global and local alignment. Furthermore, a text-image fusion model generates accurate fine-grained image-text pairs, making the supervision signal during model training more granular. Additionally, from the perspective of unifying global and local alignment, this application proposes a design utilizing self-learning token components and bidirectional mask recovery, effectively unifying global and local alignment.

[0119] Exemplary devices

[0120] Accordingly, this application also provides a target recognition device.

[0121] Please see Figure 8 In one exemplary embodiment, a target recognition device 800 is provided, comprising:

[0122] The acquisition module 810 is used to acquire first text description information of the object to be identified and multiple images to be detected. The first text description information contains descriptions of multiple parts of the object to be identified.

[0123] The input module 820 is used to input the first text description information and multiple images to be detected into the recognition model to obtain the target image output by the recognition model. The target image is the image to be detected containing the object to be identified. The recognition model has the alignment function of the local image features and local text features of the object to be identified.

[0124] In some embodiments, the target identification device 800 is further configured to:

[0125] Multiple training samples are obtained. The training samples include the region images of multiple parts of the sample object in the training image corresponding to the sample object, the second text description information of the training image, and the third text description information corresponding to each region image.

[0126] The preset model is trained based on each training sample to obtain the recognition model. The preset model learns the alignment of local image features and local text features based on each region image, each third text description information and the second text description information in the training samples.

[0127] In some embodiments, the preset model includes an image feature encoder, a text feature encoder, and a local feature alignment module, wherein...

[0128] An image feature encoder is used to encode features of images in various regions to obtain multiple local image feature vectors.

[0129] A text feature encoder is used to encode the features of the second text description information to obtain the first text feature vector, and to encode the features of each third text description information to obtain each second text feature vector;

[0130] The local feature alignment module is used to learn the alignment of local image features and local text features based on each local image feature vector, the first text feature vector, and each second text feature vector.

[0131] In some embodiments, the training samples further include a mask image corresponding to each region image and a mask text corresponding to each third text description. The preset model further includes a global feature alignment module, wherein...

[0132] The image feature encoder is also used to encode features for each masked image to obtain multiple masked image feature vectors, and to encode features for the training image to obtain a global image feature vector;

[0133] The text feature encoder is also used to perform feature encoding on each masked text to obtain multiple masked text feature vectors;

[0134] The global feature alignment module is used to perform alignment learning from local features to global features based on the global image feature vector, the feature vectors of each mask image, each mask text, and the first text feature vector. The learning results of the global feature alignment module and the learning results of the local feature alignment module are used to update the network parameters of the preset model.

[0135] In some embodiments, the global feature alignment module includes a mask image submodule and a mask text submodule, wherein...

[0136] The mask image submodule is used to recover the masked areas in the mask image corresponding to the feature vector of the mask image;

[0137] The masked text submodule is used to recover the hidden text in the masked text corresponding to the masked text feature vector.

[0138] In some embodiments, the local feature alignment module is used to concatenate the first text feature vector and each of the second text feature vectors to obtain a concatenated feature vector, extract features from the concatenated feature vector to obtain multiple sub-feature vectors, and perform alignment learning between local image features and local text features based on each sub-feature vector and each local feature vector.

[0139] In some embodiments, the target identification device 800 is further configured to:

[0140] The training image of the sample object is input into the text image fusion model to obtain the second text description information, each third text description information, the mask image corresponding to the region image of multiple parts of the sample object in the training image, and the mask text corresponding to each third text description information output by the text image fusion model.

[0141] Based on the training images, the second text description information, each third text description information, each mask image, and each mask text, construct the training samples corresponding to the training images.

[0142] The target recognition device provided in this embodiment belongs to the same concept as the target recognition method provided in the above embodiments of this application. It can execute the target recognition method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the target recognition method provided in the above embodiments of this application, and will not be repeated here.

[0143] Exemplary electronic devices

[0144] Another embodiment of this application also proposes an electronic device. See [link to relevant documentation]. Figure 9 As shown, the electronic device includes a memory 900 and a processor 910; wherein the memory 900 is connected to the processor 910 and is used to store programs; the processor 910 is used to implement the target recognition method disclosed in any of the above embodiments by running the programs stored in the memory 900.

[0145] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 920, an input device 930, and an output device 940.

[0146] The processor 910, memory 900, communication interface 920, input device 930, and output device 940 are interconnected via a bus. Among them:

[0147] A bus can include a pathway for transmitting information between various components of a computer system.

[0148] The processor 910 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0149] The processor 910 may include a main processor, as well as a baseband chip, modem, etc.

[0150] The memory 900 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 900 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0151] Input device 930 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0152] Output device 940 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0153] The communication interface 920 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0154] The processor 910 executes the program stored in the memory 900 and calls other devices, which can be used to implement the various steps of any of the target recognition methods provided in the above embodiments of this application.

[0155] This application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the target recognition method described in any of the above embodiments. For details of the processing and its beneficial effects, please refer to the above embodiments of the target recognition method.

[0156] Exemplary computer program products and storage media

[0157] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the target recognition methods according to various embodiments of this application as described in any of the above embodiments of this specification.

[0158] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0159] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor to perform the steps of the target recognition method according to various embodiments of this application as described in any of the above embodiments of this specification.

[0160] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0161] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0162] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0163] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0164] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0165] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0166] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0167] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0168] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0169] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0170] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target recognition method, characterized in that, include: Acquire first textual description information of the object to be identified and multiple images to be detected, wherein the first textual description information contains descriptions of multiple parts of the object to be identified; The first text description information and multiple images to be detected are input into the recognition model to obtain the target image output by the recognition model. The target image is an image to be detected containing the object to be identified. The recognition model has the function of aligning the local image features and local text features of the object to be identified.

2. The target recognition method according to claim 1, characterized in that, Before obtaining the first textual description information of the object to be identified and multiple images to be detected, the method further includes: Multiple training samples are obtained, wherein the training samples include region images of multiple parts of a sample object in the training image corresponding to the sample object, second textual description information of the training image, and third textual description information corresponding to each region image; The preset model is trained based on each of the training samples to obtain the recognition model, wherein the preset model learns the alignment of local image features and local text features based on each of the region images, each of the third text description information and the second text description information in the training samples.

3. The target recognition method according to claim 2, characterized in that, The preset model includes an image feature encoder, a text feature encoder, and a local feature alignment module, wherein... The image feature encoder is used to encode features of each of the region images to obtain multiple local image feature vectors; The text feature encoder is used to perform feature encoding on the second text description information to obtain a first text feature vector, and to perform feature encoding on each of the third text description information to obtain each of the second text feature vectors; The local feature alignment module is used to perform alignment learning of local image features and local text features based on each of the local image feature vectors, the first text feature vector, and each of the second text feature vectors.

4. The target recognition method according to claim 3, characterized in that, The training samples also include a mask image corresponding to each of the said region images and a mask text corresponding to each of the said third text description information. The preset model also includes a global feature alignment module, wherein... The image feature encoder is further configured to perform feature encoding on each of the mask images to obtain multiple mask image feature vectors, and to perform feature encoding on the training images to obtain a global image feature vector; The text feature encoder is also used to perform feature encoding on each of the masked texts to obtain multiple masked text feature vectors; The global feature alignment module is used to perform alignment learning from local features to global features based on the global image feature vector, each of the mask image feature vectors, each of the mask texts, and the first text feature vector. The learning results of the global feature alignment module and the learning results of the local feature alignment module are used to update the network parameters of the preset model.

5. The target recognition method according to claim 4, characterized in that, The global feature alignment module includes a mask image submodule and a mask text submodule, wherein, The mask image submodule is used to recover the masked area in the mask image corresponding to the feature vector of the mask image; The masked text submodule is used to recover the masked text that is hidden in the masked text corresponding to the masked text feature vector.

6. The target recognition method according to claim 3, characterized in that, The local feature alignment module performs alignment learning of local image features and local text features based on each of the local image feature vectors, the first text feature vector, and each of the second text feature vectors, including: The first text feature vector and each of the second text feature vectors are concatenated to obtain a concatenated feature vector. The concatenated feature vector is subjected to feature extraction to obtain multiple sub-feature vectors; Alignment learning between local image features and local text features is performed based on each of the sub-feature vectors and each of the local feature vectors.

7. The target recognition method according to any one of claims 2-6, characterized in that, The training samples also include a mask image corresponding to each of the region images and a mask text corresponding to each of the third text description information. Obtaining multiple training samples includes: The training image of the sample object is input into the text image fusion model to obtain the second text description information, each of the third text description information, the mask image corresponding to the region image of the multiple parts of the sample object in the training image, and the mask text corresponding to each of the third text description information output by the text image fusion model. Training samples corresponding to the training images are constructed based on the training images, the second text description information, each of the third text description information, each of the mask images, and each of the mask texts.

8. A target recognition device, characterized in that, include: The acquisition module is used to acquire first textual description information of the object to be identified and multiple images to be detected. The first textual description information includes descriptions of multiple parts of the object to be identified. The input module is used to input the first text description information and multiple images to be detected into the recognition model to obtain the target image output by the recognition model. The target image is an image to be detected containing the object to be identified. The recognition model has the alignment function of the local image features and local text features of the object to be identified.

9. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the target recognition method as described in any one of claims 1 to 7 by running a program in the memory.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the target recognition method as described in any one of claims 1 to 7.

Images