An image retrieval method, apparatus and device

Abstract

The application provides an image retrieval method, device and equipment, which comprises the following steps: obtaining an original image of a target object and a text prompt word, the text prompt word being used to indicate a target attribute dimension of the original image; adjusting the image content corresponding to the target attribute dimension in the original image based on the text prompt word to obtain a multi-dimensional image of the target object; constructing a sample image set, the sample image set comprising a plurality of sample images of each target object, the plurality of sample images comprising the original image of the target object and the multi-dimensional image of the target object; adjusting the network parameters of an initial image encoder to be trained based on the sample image set to obtain a trained target image encoder; wherein the target image encoder is used to determine whether the object in a first to-be-detected image and the object in a second to-be-detected image are the same object. Through the technical scheme of the application, the same object can be accurately identified, and a more reliable pedestrian re-identification model can be obtained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image retrieval method, apparatus and device. Background Technology

[0002] Person re-identification (ReID) algorithms are computer vision technologies used to identify pedestrians across different cameras, and are widely used in smart cities and other fields. Due to differences between different cameras, appearance can be affected by clothing, scale, occlusion, posture, and viewing angle, making person re-identification algorithms a key technology in computer vision. By extracting global and local features such as clothing, body shape, and hairstyle, person re-identification algorithms can retrieve the same pedestrian in cross-camera scenes, solving the problem of identity loss caused by resolution differences and occlusion, and enabling the identification of the same pedestrian in cross-camera scenarios.

[0003] However, in real-world scenarios, various complex conditions, such as posture (standing, sitting, lying down), changing clothes and accessories, human perspective (front, back, side), camera perspective (looking down, looking up, looking straight ahead), and human figure being occluded, can cause pedestrian re-identification algorithms to fail to identify the same pedestrian. Summary of the Invention

[0004] This application provides an image retrieval method, the method comprising:

[0005] The original image and text prompts of the target object are obtained, wherein the text prompts are used to indicate the target attribute dimension of the original image; the image content in the original image corresponding to the target attribute dimension is adjusted based on the text prompts to obtain a multi-dimensional image of the target object;

[0006] Construct a sample image set, which includes multiple sample images of each target object, including the original image of the target object and a multi-dimensional image of the target object;

[0007] The network parameters of the initial image encoder to be trained are adjusted based on the sample image set to obtain the trained target image encoder; wherein, the target image encoder is used to retrieve whether the object in the first image to be detected and the object in the second image to be detected are the same object.

[0008] This application provides an image retrieval device, the device comprising:

[0009] An acquisition module is used to acquire the original image and text prompts of the target object, wherein the text prompts are used to indicate the target attribute dimension of the original image; adjust the image content in the original image corresponding to the target attribute dimension based on the text prompts to obtain a multi-dimensional image of the target object; construct a sample image set, wherein the sample image set includes multiple sample images of each target object, wherein the multiple sample images include the original image of the target object and the multi-dimensional image of the target object;

[0010] The training module is used to adjust the network parameters of the initial image encoder to be trained based on the sample image set to obtain the trained target image encoder; wherein, the target image encoder is used to retrieve whether the object in the first image to be detected and the object in the second image to be detected are the same object.

[0011] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the image retrieval method of the example described above.

[0012] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image retrieval method of the example above.

[0013] This application provides a machine-readable storage medium storing machine-executable instructions that can be executed by a processor; wherein the processor is configured to execute the machine-executable instructions, and when the machine-executable instructions are executed, to implement the image retrieval method of the example described above.

[0014] As can be seen from the above technical solutions, in this embodiment, the image content corresponding to the target attribute dimension in the original image is adjusted based on text prompts to obtain multi-dimensional images of the target object, such as multi-dimensional images of different postures (standing, sitting, lying down), different clothing and accessories, different human perspectives (front, back, side), and different camera perspectives (top, bottom, eye level). After training the target image encoder based on these multi-dimensional images, it is possible to retrieve whether the objects in the image to be detected are the same object (such as pedestrians) based on the target image encoder, thereby accurately identifying the same object. In real-world usage scenarios, even with various complex conditions such as posture, clothing and accessories, human perspectives, camera perspectives, and human occlusion, the same object can still be accurately identified. Training the target image encoder based on clothing-changing multi-posture data can improve the robustness and accuracy of the target image encoder in clothing-changing multi-posture scenarios, enhance feature extraction capabilities, and help to better extract features of various attributes such as clothing, posture, age, and gender, resulting in a more reliable pedestrian re-identification model and achieving cross-modal pedestrian re-identification. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an image retrieval method according to one embodiment of this application;

[0016] Figure 2 This is a flowchart illustrating an image retrieval method according to one embodiment of this application;

[0017] Figure 3A This is a schematic diagram of the original image in one embodiment of this application;

[0018] Figure 3B This is a schematic diagram of a posture skeleton diagram in one embodiment of this application;

[0019] Figure 3C This is a schematic diagram of a multi-dimensional image in one embodiment of this application;

[0020] Figure 3D This is a schematic diagram of a multi-dimensional image in one embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the pedestrian re-identification detection model in one embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of an image retrieval device according to one embodiment of this application;

[0023] Figure 6 This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0024] This application proposes an image retrieval method that can be applied to electronic devices, such as cameras or management devices (e.g., personal computers, servers, smartphones, laptops, etc.). See [link to relevant documentation]. Figure 1 The diagram shown is a flowchart of the image retrieval method, which may include:

[0025] Step 101: Obtain the original image and text prompt of the target object. The text prompt is used to indicate the target attribute dimension of the original image. Based on the text prompt, adjust the image content in the original image corresponding to the target attribute dimension to obtain a multi-dimensional image of the target object.

[0026] Step 102: Construct a sample image set, which includes multiple sample images of each target object. The multiple sample images include the original image of the target object and the multi-dimensional image of the target object.

[0027] Step 103: Adjust the network parameters of the initial image encoder to be trained based on the sample image set to obtain the trained target image encoder; wherein, the target image encoder is used to retrieve whether the object in the first image to be detected and the object in the second image to be detected are the same object.

[0028] For example, the target attribute dimension may include, but is not limited to, at least one of clothing attribute dimension, posture attribute dimension, background attribute dimension, and person attribute dimension; adjusting the image content in the original image corresponding to the target attribute dimension based on text prompts to obtain a multi-dimensional image of the target object may include, but is not limited to: if the target attribute dimension includes the clothing attribute dimension, then adjusting the image content in the original image corresponding to the clothing attribute based on text prompts to obtain a multi-dimensional image; if the target attribute dimension includes the background attribute dimension, then adjusting the image content in the original image corresponding to the background attribute based on text prompts to obtain a multi-dimensional image; if the target attribute dimension includes the person attribute dimension, then adjusting the image content in the original image corresponding to the person attribute based on text prompts to obtain a multi-dimensional image; if the target attribute dimension includes the posture attribute dimension, then adjusting the image content in the original image corresponding to the posture attribute based on text prompts and a posture skeleton diagram to obtain a multi-dimensional image.

[0029] For example, adjusting the network parameters of the initial image encoder to be trained based on a sample image set to obtain a trained target image encoder may include, but is not limited to: inputting sample images from the sample image set into the initial image encoder to obtain image features and identity features; inputting the sample images into the initial prompt word learning network to be trained to obtain a first prompt word, and inputting the first prompt word into the text encoder to obtain a first text feature; determining a first target loss value based on the image features and the first text feature; and adjusting the network parameters of the initial prompt word learning network based on the first target loss value to obtain a trained target prompt word learning network. Furthermore, sample images may be input into the target prompt word learning network to obtain a second prompt word, and the second prompt word may be input into the text encoder to obtain a second text feature; a second target loss value may be determined based on the image features, the second text feature, and the identity feature; and the network parameters of the initial image encoder may be adjusted based on the second target loss value to obtain the target image encoder.

[0030] For example, determining the first target loss value based on image features and first text features may include, but is not limited to: determining the first loss value corresponding to the i-th sample image based on the fusion feature between the image features of the i-th sample image and the first text feature of the i-th sample image in the sample image set, and the fusion feature between the image features of the i-th sample image and the first text feature of each sample image; determining the second loss value corresponding to the i-th sample image based on the fusion feature between the image features of the i-th sample image and the first text feature of the i-th sample image, and the fusion feature between the first text feature of the i-th sample image and the image features of each sample image; determining the third loss value of the i-th sample image based on the first loss value and the second loss value, and determining the first target loss value based on the third loss value of each sample image.

[0031] For example, for each sample image in the sample image set, the identity features of the sample image may include the predicted probability value of each category supported by the initial image encoder, and the category corresponding to the maximum predicted probability value can be used as the target category of the sample image. Based on this, a second target loss value is determined based on image features, second text features, and identity features. This may include, but is not limited to: determining a fourth loss value based on the predicted probability value of the i-th sample image under each category; determining a fifth loss value for the i-th sample image based on a first distance value and a second distance value, where the first distance value is the distance between the image features of the i-th sample image and the image features of the first class of sample images, and the second distance value is the distance between the image features of the i-th sample image and the image features of the second class of sample images; wherein the target category of the first class of sample images is the same as the target category of the i-th sample image, and the target category of the second class of sample images is different from the target category of the i-th sample image; determining the category features of the category based on the second text features of the sample image corresponding to the category; determining a sixth loss value based on the fusion features between the image features of the i-th sample image and the category features of each category, and the fusion features between the image features of the i-th sample image and the second text features of each sample image; determining a seventh loss value for the i-th sample image based on the fourth, fifth, and sixth loss values; and determining a second target loss value based on the seventh loss value of each sample image.

[0032] For example, before determining the fourth loss value based on the predicted probability value of the i-th sample image under each category, it may further include: determining the category parameter value of the i-th sample image based on the maximum predicted probability value corresponding to the i-th sample image and the maximum predicted probability value corresponding to each sample image.

[0033] The fourth loss value is determined based on the predicted probability value of the i-th sample image under each category, which may include, but is not limited to: determining the fourth loss value of the i-th sample image based on the predicted probability value of the i-th sample image under each category and the category parameter value of the i-th sample image.

[0034] The sixth loss value is determined based on the fusion features between the image features of the i-th sample image and the category features of each category, and the fusion features between the image features of the i-th sample image and the second text features of each sample image. This may include, but is not limited to, the following: determining the sixth loss value based on the fusion features between the image features of the i-th sample image and the category features of each category, the fusion features between the image features of the i-th sample image and the second text features of each sample image, and the category parameter value of the i-th sample image.

[0035] For example, the initial cue word learning network or the target cue word learning network may include a text template that includes multiple attribute dimensions. The text template is used to instruct the initial cue word learning network or the target cue word learning network to output cue words with multiple attribute dimensions. The multiple attribute dimensions may include clothing attribute dimensions, posture attribute dimensions, background attribute dimensions, and character attribute dimensions.

[0036] For example, adjusting the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network may include: freezing the network parameters of the text encoder and the initial image encoder, and adjusting the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network. Adjusting the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder may include: freezing the network parameters of the text encoder and the target prompt word learning network, and adjusting the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder.

[0037] As can be seen from the above technical solutions, in this embodiment, the image content corresponding to the target attribute dimension in the original image is adjusted based on text prompts to obtain multi-dimensional images of the target object, such as multi-dimensional images of different postures (standing, sitting, lying down), different clothing and accessories, different human perspectives (front, back, side), and different camera perspectives (top, bottom, eye level). After training the target image encoder based on these multi-dimensional images, it is possible to retrieve whether the objects in the image to be detected are the same object (such as pedestrians) based on the target image encoder, thereby accurately identifying the same object. In real-world usage scenarios, even with various complex conditions such as posture, clothing and accessories, human perspectives, camera perspectives, and human occlusion, the same object can still be accurately identified. Training the target image encoder based on clothing-changing multi-posture data can improve the robustness and accuracy of the target image encoder in clothing-changing multi-posture scenarios, enhance feature extraction capabilities, and help to better extract features of various attributes such as clothing, posture, age, and gender, resulting in a more reliable pedestrian re-identification model and achieving cross-modal pedestrian re-identification.

[0038] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0039] Pedestrian re-identification (ReID) algorithms can be applied to outdoor and home detection scenarios. They can detect target objects (such as pedestrians) across cameras and then perform behavior analysis on the target objects.

[0040] However, in real-world scenarios, various complex conditions, such as posture (standing, sitting, lying down), changing clothes and accessories, human perspective (front, back, side), camera perspective (looking down, looking up, looking straight ahead), and human figure being occluded, can cause pedestrian re-identification algorithms to fail to identify the same target object.

[0041] In response to the above findings, this application proposes a cross-modal robust pedestrian re-identification method based on multi-pose data of changing clothes. This method can adjust the image content in the original image corresponding to the target attribute dimension based on text prompts to obtain a multi-dimensional image of the target object, thereby amplifying samples such as human attributes and clothing. The generated samples are richer, the effect is better, and the robustness of the model is enhanced.

[0042] This application proposes an image retrieval method applicable to product fields such as pedestrian re-identification algorithms, including intelligent detection and smart homes. This method can also be applied to electronic devices, such as cameras or management equipment. See also... Figure 2 The diagram shown is a flowchart of the method, which may include:

[0043] Step 201: Obtain the original image and text prompt of the target object. The text prompt is used to indicate the target attribute dimension of the original image. The target attribute dimension may include, but is not limited to, at least one of the following: clothing attribute dimension, posture attribute dimension, background attribute dimension, and person attribute dimension.

[0044] For example, the original image may be an image captured by a camera, and the original image may include an object (such as a pedestrian object), which may be referred to as the target object.

[0045] For example, text prompts can be pre-configured. These prompts can be configured individually for each target object, or the same prompt can be configured for all target objects. In this embodiment, targeted prompts can be designed for home smart detection scenarios. For instance, the text prompt is used to indicate the target attribute dimension, which may include, but is not limited to, at least one of the following: clothing attribute dimensions (such as color, style, season), posture attribute dimensions (such as walking, standing, occlusion), background attribute dimensions (such as living room, kitchen, corridor, yard), and person attribute dimensions (such as age, gender).

[0046] For example, text prompts can include clothing attributes, such as "Please generate an image of a red top," "Please generate an image of a summer top," or "Please generate an image of a windbreaker." Text prompts can also include posture attributes, such as "Please generate an image of a walking posture," "Please generate an image of a standing posture," or "Please generate an image of a sitting posture." Text prompts can also include background attributes, such as "Please generate an image of a living room," "Please generate an image of a kitchen," or "Please generate an image of a hallway." Finally, text prompts can include person attributes, such as "Please generate an image of someone who is 18 years old." Text prompts can include one or more of the above attributes; there is no limitation on the type of text prompt.

[0047] For example, in the above method, the settings for the generated scene and generated prompts are more targeted, and it is possible to generate clothing change data and human images in different poses in the scene.

[0048] Step 202: Based on the text prompt, adjust the image content (i.e., image part / image region) in the original image corresponding to the target attribute dimension to obtain a multi-dimensional image of the target object.

[0049] For example, if the target attribute dimension includes a clothing attribute dimension, the image content in the original image corresponding to the clothing attribute is adjusted based on the text prompt to obtain a multi-dimensional image. For instance, if the text prompt is "Please generate an image of a red top," and the image content in the original image corresponding to the clothing attribute is the top area of ​​the target object, then adjusting the top area of ​​the target object in the original image yields a multi-dimensional image of a red top (the top area of ​​the target object is a red top).

[0050] For example, if the target attribute dimension includes a background attribute dimension, the image content in the original image corresponding to the background attribute is adjusted based on the text prompt to obtain a multi-dimensional image. For instance, if the text prompt is "Please generate an image of a living room," the image content in the original image corresponding to the background attribute is the background area of ​​the target object. Therefore, the background area of ​​the target object in the original image is adjusted to obtain a multi-dimensional image of the living room background (the background area of ​​the target object is the living room).

[0051] For example, if the target attribute dimension includes a person attribute dimension, the image content in the original image corresponding to the person attribute is adjusted based on the text prompt to obtain a multi-dimensional image. For instance, if the text prompt is "Please generate an image of an 18-year-old," the image content in the original image corresponding to the person attribute is the target object itself. Therefore, the target object in the original image is adjusted, that is, the age of the target object is adjusted to 18 years old, resulting in a multi-dimensional image of an 18-year-old target object.

[0052] For example, if the target attribute dimension includes a pose attribute dimension, the image content corresponding to the pose attribute in the original image is adjusted based on the text prompt and the pose skeleton diagram to obtain a multi-dimensional image. For instance, if the text prompt is "Please generate an image of a standing pose," and the image content corresponding to the person attribute in the original image is the target object itself, then the pose of the target object in the original image is adjusted to a standing pose, resulting in a multi-dimensional image of the standing target object. Regarding how to adjust the pose of the target object in the original image, one can refer to the pose skeleton diagram of a standing pose; that is, the standing target object is obtained based on this pose skeleton diagram.

[0053] In one possible implementation, a multi-dimensional image of the target object can be generated using a Stable Diffusion model. The Stable Diffusion model can be applied to image processing tasks such as image denoising, image segmentation, image enhancement, and image restoration. For example, a text prompt and the original image can be input into the Stable Diffusion model. The model can then adjust the image content in the original image corresponding to the target attribute dimension based on the text prompt to obtain a multi-dimensional image of the target object. This embodiment does not limit the processing procedure of the Stable Diffusion model.

[0054] For the Stable Diffusion model, the input data consists of text prompts and original images (such as high-resolution human images). The output data consists of multiple multi-dimensional images of the target object, each corresponding to the same identity ID (i.e., the identity ID of the same target object). These multi-dimensional images can include images of different clothing items and images of different poses. For example, multi-dimensional images could include multi-dimensional images of a red top, multi-dimensional images of a summer top, multi-dimensional images of a walking pose, multi-dimensional images of a red top and a walking pose, and multi-dimensional images of the object in a living room, etc.

[0055] For the Stable Diffusion model, the generation stability of the Stable Diffusion model can be improved by controlling the noise scheduling parameters and the number of sampling steps. There are no restrictions on this process.

[0056] If the text prompts involve clothing, background, and character attributes, but not pose, then the input data for the Stable Diffusion model is the text prompts and the original image. The output data of the Stable Diffusion model is multiple multi-dimensional images of the target object. In this way, using the original image and text prompts, multi-dimensional images with different clothing, backgrounds, and character attributes can be generated.

[0057] If the text prompt involves the pose attribute dimension, the input data for the Stable Diffusion model consists of the text prompt, the original image, and a pose skeleton map. The pose skeleton map can reflect the skeleton of a face in various poses. The output data of the Stable Diffusion model is multiple multi-dimensional images of the target object. In this way, multi-dimensional images with different poses can be generated using the original image, text prompt, and pose skeleton map.

[0058] For example, see Figure 3A The image shown is a schematic diagram of the original image. (See attached image.) Figure 3B The image shown is a schematic diagram of a pose skeleton (a pose skeleton diagram of a standing posture). By inputting the original image, text prompts, and pose skeleton diagram into the StableDiffusion model, a multi-dimensional image can be obtained. See [link / reference]. Figure 3C The image shown is a schematic diagram of a multi-dimensional image. Figure 3C The original image and text prompts are input into the Stable Diffusion model to obtain a multi-dimensional image. See [link / reference]. Figure 3D The image shown is a schematic diagram of a multi-dimensional image.

[0059] In summary, this embodiment generates multi-dimensional images of changing clothes and multiple poses through a stable diffusion model, amplifying human-shaped attributes, clothing, and other samples. It focuses on enhancing and amplifying human-shaped samples, resulting in better stability and controllability, better image quality, and support for high-resolution image generation.

[0060] Step 203: Construct a sample image set, which includes multiple sample images of each target object. The multiple sample images include the original image of the target object and the multi-dimensional image of the target object.

[0061] For example, for target object a1, steps 201 and 202 can obtain the original image and multi-dimensional images (multiple multi-dimensional images) of target object a1. Thus, the sample image set includes multiple sample images of target object a1, namely, the original image and multiple multi-dimensional images of target object a1. These sample images can be automatically labeled with ID tags and attribute tags. The ID tags of these sample images are all identifiers of target object a1, representing sample images of target object a1. The attribute tags of these sample images can be clothing, posture, age, background, etc., and can be determined based on text prompts; there are no restrictions on this.

[0062] For target object a2, the sample image set includes multiple sample images of target object a2, namely the original image of target object a2 and multiple multi-dimensional images. These sample images can be automatically labeled with ID tags (identifiers of target object a2) and attribute tags. Similarly, assuming there are 50 target objects, the sample image set includes sample images of all 50 target objects, with multiple sample images corresponding to each target object.

[0063] In one possible implementation, for each target object, after obtaining the original image and multiple multi-dimensional images of the target object, the original image of the target object can be used as a sample image. For the multiple multi-dimensional images of the target object, these multi-dimensional images can be filtered, with multi-dimensional images that meet the requirements being used as sample images, and those that do not meet the requirements not being used as sample images.

[0064] For example, for each multi-dimensional image, its image quality can be determined, such as using a NIMA (Neural Image Assessment) model. If the image quality is greater than a quality threshold, the multi-dimensional image is used as a sample image; if the image quality is not greater than the quality threshold, it is not used as a sample image. In this way, low-quality multi-dimensional images can be filtered out, and high-quality multi-dimensional images can be used as sample images.

[0065] For example, for each multi-dimensional image, the similarity between the multi-dimensional image and the original image can be determined. If the similarity is greater than a similarity threshold, the multi-dimensional image is used as a sample image; if the similarity is not greater than the similarity threshold, the multi-dimensional image is not used as a sample image.

[0066] For example, for each multi-dimensional image, its image quality can be determined. If the image quality is not greater than a quality threshold, the multi-dimensional image is not used as a sample image. If the image quality is greater than the quality threshold, instead of directly using the multi-dimensional image as a sample image, the similarity between the multi-dimensional image and the original image is determined. If the similarity is greater than a similarity threshold, the multi-dimensional image is used as a sample image; otherwise, it is not used as a sample image.

[0067] Step 204: Obtain the pedestrian re-identification detection model, which can be denoted as the CLIP model.

[0068] For example, a person re-identification detection model may include an initial cue word learning network, an initial image encoder, and a text encoder, see [link to relevant documentation]. Figure 4 The diagram shown is a structural schematic of the pedestrian re-identification detection model. This pedestrian re-identification detection model is the model to be trained, and subsequent training will be performed on the pedestrian re-identification detection model.

[0069] For example, the input data of the learnable token network can be sample images, and the output data of the learnable token network is the token. There are no restrictions on the network structure of the learnable token network, as long as the learnable token network can extract the tokens from the sample images.

[0070] For example, the initial cue word learning network may include a text template, i.e., a text template is pre-configured in the initial cue word learning network. This text template includes multiple attribute dimensions, and is used to instruct the initial cue word learning network to output cue words with multiple attribute dimensions. Among these multiple attribute dimensions, there may be, but are not limited to, clothing attribute dimensions, posture attribute dimensions, background attribute dimensions, and character attribute dimensions.

[0071] Based on this, the initial cue word learning network can extract cue words corresponding to the clothing attribute dimension, cue words corresponding to the posture attribute dimension, cue words corresponding to the background attribute dimension, and cue words corresponding to the person attribute dimension from sample images.

[0072] For example, the input data (i.e., cue words) of the text encoder of the person re-identification and detection model can be designed, and the input data of the text encoder is the output data of the initial cue word learning network. The design of the cue words follows the attribute dimensions that are of concern when generating multi-dimensional images, such as clothing attribute dimensions, pose attribute dimensions, background attribute dimensions, and person attribute dimensions (age, gender), etc. In this way, a text template can be designed and pre-configured in the initial cue word learning network. For example, an example of this text template can be: A {age} {gender} {pose} person wearing{clothing} in {scene}. Here, the words in {} are the cue words (tokens) that the initial cue word learning network needs to extract.

[0073] Based on this, the initial cue word learning network can extract "age" cue words, "gender" cue words, "posture" cue words, "clothing" cue words, and "background" cue words.

[0074] For example, the input data of the initial image encoder can be a sample image, and the output data of the initial image encoder can be image features and identity features. There are no restrictions on the network structure of the initial image encoder, as long as the initial image encoder can extract the image features and identity features of the sample image.

[0075] For example, the input data of the text encoder can be cue words (i.e., the output data of the initial cue word learning network), and the output data of the text encoder can be text features. There are no restrictions on the network structure of the text encoder, as long as the text encoder can generate text features based on the cue words.

[0076] In summary, image features of sample images can be extracted through an initial image encoder, and text features of sample images (text features generated based on prompts in the sample images) can be extracted through a text encoder. Then, the image features and text features can be associated to build a cross-modal feature association.

[0077] Step 205: Input the sample images (multiple sample images) from the sample image set into the initial image encoder to obtain image features and identity features; input the sample image into the initial prompt word learning network to obtain the first prompt word, and input the first prompt word into the text encoder to obtain the first text feature.

[0078] For example, the initial image encoder may include a first sub-network and a second sub-network. A sample image can be input to the first sub-network, which processes it to obtain image features corresponding to the sample image. This processing procedure is not limited; the first sub-network only needs to be able to extract image features. Further, after obtaining the image features, they can be input to the second sub-network, which processes them to obtain identity features corresponding to the sample image. This processing procedure is also not limited; the second sub-network only needs to be able to extract identity features. Alternatively, the sample image can be input to the first sub-network, which processes it to obtain image features corresponding to the sample image. The sample image can also be input to the second sub-network, which processes it to obtain identity features corresponding to the sample image.

[0079] For the identity features corresponding to the sample image, these features can include the predicted probability values ​​for each category supported by the initial image encoder, and the category corresponding to the highest predicted probability value can be used as the target category of the sample image. For example, assuming that sample images of 50 target objects are input into the initial image encoder, the initial image encoder supports 50 categories: category 1 corresponds to target object a1, category 2 corresponds to target object a2, ..., category 5 corresponds to target object a50. Based on this, the identity feature can be a 50-dimensional probability vector representing the predicted probability values ​​of the 50 categories. Assuming that the predicted probability value of category 3 is the highest predicted probability value, then category 3 can be used as the target category of the sample image.

[0080] For ease of description, taking the initial image encoder supporting 4 categories as an example, the identity feature (i.e., probability vector) can be [0.8, 0.1, 0.07, 0.03]. 0.8 represents the predicted probability value that the sample image belongs to category 1 (i.e., the sample image belongs to target object a1), 0.1 represents the predicted probability value that the sample image belongs to category 2, 0.07 represents the predicted probability value that the sample image belongs to category 3, and 0.03 represents the predicted probability value that the sample image belongs to category 4. Obviously, 0.8 is the largest, and the target category is category 1.

[0081] For example, a sample image can be input into an initial cue word learning network. The initial cue word learning network processes the sample image to obtain the first cue word. This processing method is not limited; the initial cue word learning network only needs to be able to extract the first cue word from the sample image. For instance, the initial cue word learning network can include a text template with multiple attribute dimensions. The initial cue word learning network needs to extract cue words corresponding to each attribute dimension from the sample image and use these cue words as the first cue word. For example, the initial cue word learning network might extract cue words corresponding to the clothing attribute dimension, such as clothing color; cue words corresponding to the posture attribute dimension, such as standing posture; cue words corresponding to the background attribute dimension, such as living room; and cue words corresponding to the person attribute dimension, such as age 18.

[0082] For example, the initial prompt word learning network outputs a first prompt word, which is then input into the text encoder. The text encoder processes the first prompt word to obtain the first text feature. There are no restrictions on this processing procedure; the text encoder only needs to be able to generate the first text feature based on the first prompt word.

[0083] Step 206: Determine the first target loss value based on image features and first text features.

[0084] In one possible implementation, the first target loss value can be determined using the following steps. This is merely an example and not a limitation, as long as the first target loss value is related to the image features and the first text features. For instance, a loss function can be pre-configured, whose inputs are the image features and the first text features, and whose output is the first target loss value. Thus, by substituting the image features and the first text features into this loss function, the first target loss value can be obtained.

[0085] Step S11: Based on the fusion features between the image features of the i-th sample image and the first text features of the i-th sample image, and the fusion features between the image features of the i-th sample image and the first text features of each sample image, determine the first loss value corresponding to the i-th sample image.

[0086] For example, the first loss value corresponding to the i-th sample image can be determined using the following formula (1):

[0087] Formula (1)

[0088] In formula (1), V represents the first loss value corresponding to the i-th sample image, where "i" represents the i-th sample image in the sample image set. i T represents the image features of the i-th sample image. i T represents the first text feature of the i-th sample image. a Let represent the first text feature of the a-th sample image, where 'a' ranges from 1 to B, and B represents the total number of sample images. This represents the fusion feature between the image features of the i-th sample image and the first text feature of the i-th sample image. When the value of 'a' ranges from 1 to B, The fusion feature represents the image feature of the i-th sample image and the first text feature of each sample image (i.e., each sample image from 1 to B in the sample image set).

[0089] Regarding the fusion feature, taking the fusion feature between the image feature of the i-th sample image and the first text feature of the i-th sample image as an example, the fusion feature can be the product of the image feature and the first text feature, or the result of the convolution operation between the image feature and the first text feature, or other methods can be used to determine the fusion feature. There are no restrictions on the method of determining this fusion feature. For example, the fusion feature can be determined using the following formula: .

[0090] Step S12: Based on the fusion features between the image features of the i-th sample image and the first text features of the i-th sample image, and the fusion features between the first text features of the i-th sample image and the image features of each sample image, determine the second loss value corresponding to the i-th sample image.

[0091] For example, the second loss value corresponding to the i-th sample image can be determined using the following formula (2):

[0092] Formula (2)

[0093] In formula (2), V represents the second loss value corresponding to the i-th sample image. i T represents the image features of the i-th sample image. i V represents the first text feature of the i-th sample image. a Let B represent the image features of the a-th sample image, and let B represent the total number of sample images. This represents the fusion feature between the image features of the i-th sample image and the first text feature of the i-th sample image. When the value of 'a' ranges from 1 to B, The fused feature represents the first text feature of the i-th sample image and the image features of each sample image (i.e., each sample image from 1 to B in the sample image set).

[0094] Step S13: Determine the third loss value of the i-th sample image based on the first loss value and the second loss value.

[0095] For example, a third loss value can be obtained by weighting the first and second loss values. Alternatively, the sum of the first and second loss values ​​can be used as the third loss value.

[0096] For example, the third loss value of the i-th sample image can be determined using the following formula (3):

[0097] Formula (3)

[0098] In formula (3), This represents the third loss value corresponding to the i-th sample image.

[0099] Step S14: Determine the first target loss value based on the third loss value of each sample image.

[0100] For example, after obtaining the third loss value for each sample image in the sample image set, the average of the third loss values ​​of all sample images can be calculated, and this average value can be used as the first target loss value.

[0101] Step 207: Adjust the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network. For example, the network parameters of the text encoder and the initial image encoder can be frozen, that is, the network parameters of the text encoder and the initial image encoder are not adjusted. Based on the frozen network parameters of the text encoder and the initial image encoder, the network parameters of the initial prompt word learning network are adjusted based on the first target loss value.

[0102] For example, when adjusting the network parameters of the initial prompt word learning network based on the first target loss value, the adjustment goal is to make the first target loss value smaller and smaller. For example, the network parameters of the initial prompt word learning network can be adjusted by using gradient descent or other methods. There are no restrictions on this adjustment process.

[0103] After adjusting the network parameters of the initial prompt word learning network based on the first target loss value, an adjusted prompt word learning network can be obtained. If the adjusted prompt word learning network has converged, it is used as the target prompt word learning network, and subsequent steps are executed. If the adjusted prompt word learning network has not converged, it is used as the initial prompt word learning network, and the process returns to step 205. The operation of inputting the sample image into the initial prompt word learning network to obtain the first prompt word, inputting the first prompt word into the text encoder to obtain the first text feature is performed, and the above steps are repeated.

[0104] For example, if the first objective loss value is less than the loss value threshold, then the adjusted prompt word learning network has converged; if the first objective loss value is not less than the loss value threshold, then the adjusted prompt word learning network has not converged.

[0105] For example, if the number of iterations reaches the threshold, the adjusted prompt word learning network has converged; if the number of iterations does not reach the threshold, the adjusted prompt word learning network has not converged.

[0106] For example, if the iteration duration reaches the duration threshold, the adjusted prompt word learning network has converged; if the iteration duration does not reach the duration threshold, the adjusted prompt word learning network has not converged.

[0107] Of course, the above are just a few examples of how to determine whether convergence has occurred, and there are no restrictions on the method of determination.

[0108] For example, after obtaining the target prompt word learning network, the first stage of training is completed. In this first stage, the target prompt word learning network can be trained unsupervised. During training, an initial image encoder can be used to extract image features (image features can also be called image semantic features), and a text encoder can be used to extract text features, without the need for manual text annotation. During training, the network parameters (weights) of both the text encoder and the initial image encoder are fixed; only the initial prompt word learning network needs to be trained.

[0109] Step 208: Input the sample images (multiple sample images in the sample image set) into the target prompt word learning network to obtain the second prompt word, and input the second prompt word into the text encoder to obtain the second text feature.

[0110] For example, a sample image can be input into a target cue word learning network, which then processes the image to obtain a second cue word. For instance, the target cue word learning network can include a text template with multiple attribute dimensions. The network then needs to extract cue words corresponding to each attribute dimension from the sample image and use these cue words as the second cue words. For example, the network might extract cue words corresponding to clothing, pose, background, and person attributes from the sample image.

[0111] For example, the target cue word learning network outputs a second cue word, which is then input into the text encoder. The text encoder processes the text based on the second cue word to obtain a second text feature.

[0112] Step 209: Determine the second target loss value based on image features, second text features, and identity features.

[0113] For example, the second target loss value can be determined using the following steps. This is just an example and is not limited, as long as the second target loss value is related to the image features, the second text features, and the identity features. For instance, a loss function can be pre-configured, whose inputs are the image features, the second text features, and the identity features, and whose output is the second target loss value. Thus, by substituting the image features, the second text features, and the identity features into this loss function, the second target loss value can be obtained.

[0114] Step S21: For each sample image in the sample image set, determine the maximum predicted probability value and the target category of the sample image based on the identity features of the sample image.

[0115] For example, referring to step 205, for the identity feature corresponding to the sample image, the identity feature may include the predicted probability value of each category supported by the initial image encoder. For example, the identity feature (i.e. the probability vector) may be [0.8, 0.1, 0.07, 0.03]. Here, the predicted probability values ​​of 4 categories are shown.

[0116] Based on this, the maximum predicted probability value corresponding to the sample image can be determined. For example, a predicted probability value of 0.8 can be considered the maximum predicted probability value. The target category of the sample image can be determined. For example, the category corresponding to the maximum predicted probability value can be used as the target category of the sample image. For example, the target category can be category 1.

[0117] Step S22: Based on the maximum predicted probability value corresponding to the i-th sample image and the maximum predicted probability value corresponding to each sample image, determine the category parameter value corresponding to the i-th sample image.

[0118] For example, the category parameter value corresponding to the i-th sample image can be determined using the following formula (4):

[0119] Formula (4)

[0120] In formula (4), This represents the category parameter value corresponding to the i-th sample image. k Let the target category of the i-th sample image be category. k . This represents the maximum predicted probability value corresponding to the i-th sample image. The value of 'a' ranges from 1 to B, where B represents the total number of sample images. This represents the maximum predicted probability value for each sample image (i.e., each sample image from 1 to B in the sample image set).

[0121] Step S23: Determine the fourth loss value of the i-th sample image based on the predicted probability value of the i-th sample image under each category. For example, determine the fourth loss value of the i-th sample image based on the predicted probability value of the i-th sample image under each category and the category parameter value of the i-th sample image.

[0122] For example, the fourth loss value of the i-th sample image can be determined using the following formula (5):

[0123] Formula (5)

[0124] In formula (5), This represents the fourth loss value for the i-th sample image. This represents the category parameter value corresponding to the i-th sample image, j represents the j-th category supported by the initial image encoder, and N represents the total number of categories supported by the initial image encoder, such as 50 categories. This represents the predicted probability value of the i-th sample image in category j. For example, when the identity feature is [0.8, 0.1, 0.07, 0.03], N is 4. This represents the predicted probability value of the i-th sample image in category 1, such as 0.8. This represents the predicted probability value of the i-th sample image in category 2, such as 0.1. This represents the predicted probability value of the i-th sample image in category 3, such as 0.07. This represents the predicted probability value of the i-th sample image in category 4, such as 0.03.

[0125] Step S24: Determine the fifth loss value for the i-th sample image based on the first distance value and the second distance value. The first distance value is the distance between the image features of the i-th sample image and the image features of the first class of sample images, and the second distance value is the distance between the image features of the i-th sample image and the image features of the second class of sample images. The target category of the first class of sample images can be the same as the target category of the i-th sample image, while the target category of the second class of sample images can be different from the target category of the i-th sample image.

[0126] For example, the fifth loss value of the i-th sample image can be determined using the following formula (6):

[0127] Formula (6)

[0128] In formula (6), This represents the fifth loss value for the i-th sample image. d p This represents the first distance value between the image features of the i-th sample image and the image features of the first class of sample images. d n Let represent the distance value between the image features of the i-th sample image and the image features of the second class of sample images. This indicates the configured parameter value, which can be configured according to actual needs. This parameter value may not be present in formula (6).

[0129] For example, based on the target category of the i-th sample image, a first-class sample image can be selected from all sample images in the sample image set. The target category of the first-class sample images is the same as that of the i-th sample image, and there can be multiple first-class sample images. Based on this, a first distance value, such as Euclidean distance, can be calculated between the image features of the i-th sample image and the image features of each first-class sample image. Clearly, for multiple first-class sample images, multiple first distance values ​​can be obtained.

[0130] Based on the target category of the i-th sample image, a second category of sample images can be selected from all sample images in the sample image set. The target category of the second category of sample images is different from that of the i-th sample image, and there can be multiple second category sample images. Obviously, the second category of sample images consists of the remaining sample images excluding the first category of sample images. Based on this, a second distance value can be calculated between the image features of the i-th sample image and the image features of each second category of sample image, and multiple second distance values ​​can be obtained.

[0131] Furthermore, the first distance value is used as... d p Take the first second distance value as d nThen substitute it into formula (6) We can obtain a candidate parameter value 1 based on the first distance value and the second distance value. Similarly, we can obtain a candidate parameter value 2 based on the first distance value and the last distance value, and so on, based on the first distance value and the last distance value. Thus, we can obtain candidate parameter values ​​for all distance value combinations (one first distance value and one second distance value). The largest candidate parameter value can then be used as the fifth loss value for the i-th sample image.

[0132] Step S25: For each category supported by the initial image encoder, determine the category features of that category based on the second text features of the sample images (such as all sample images) corresponding to that category.

[0133] For example, the category features of category 1 can be determined based on the second text features of all sample images corresponding to category 1 (i.e., the target category of these sample images can be category 1). For example, the average feature of the second text features of these sample images can be used as the category feature of category 1, the maximum feature of the second text features of these sample images can be used as the category feature of category 1, or the second text features of these sample images can be fused to obtain the category features of category 1, or the second text features of these sample images can be input into a neural network model to obtain the category features of category 1. There are no restrictions on this.

[0134] The category features of category 2 are determined based on the second text features of all sample images corresponding to category 2. By analogy, the category features of each category supported by the initial image encoder can be obtained.

[0135] Step S26: Determine the sixth loss value of the i-th sample image based on the fusion features between the image features of the i-th sample image and the category features of each category, and the fusion features between the image features of the i-th sample image and the second text features of each sample image. For example, when determining the sixth loss value of the i-th sample image, the sixth loss value of the i-th sample image can be determined based on the fusion features between the image features of the i-th sample image and the category features of each category, the fusion features between the image features of the i-th sample image and the second text features of each sample image, and the category parameter value of the i-th sample image.

[0136] For example, the sixth loss value of the i-th sample image can be determined using the following formula (7):

[0137] Formula (7)

[0138] In formula (7), This represents the sixth loss value for the i-th sample image. V represents the category parameter value corresponding to the i-th sample image. i This represents the image features of the i-th sample image. Indicates category k Category characteristics, k This represents the k-th category supported by the initial image encoder, and N represents the total number of categories supported by the initial image encoder. When k is 1, This represents the category feature of category 1, when k is 2. This represents the category feature of category 2, and so on. Clearly, the value of k ranges from 1 to N. This represents the fusion feature between the image features of the i-th sample image and the category features of each category (i.e., each category from category 1 to category N). The method for determining the fusion feature is described in step S11.

[0139] In formula (7), Indicates the first The second text feature of each sample image The value range is from 1 to B, where B represents the total number of sample images. When it is 1, The second text feature representing the first sample image, in When it is 2, This represents the second text feature of the second sample image, and so on. Clearly, This represents the fusion feature between the image features of the i-th sample image and the second text features of each sample image. The method for determining the fusion feature can be found in step S11.

[0140] Step S28: Determine the seventh loss value for the i-th sample image based on the fourth, fifth, and sixth loss values. For example, the seventh loss value can be obtained by weighting the fourth, fifth, and sixth loss values. Alternatively, the sum of the fourth, fifth, and sixth loss values ​​can be used as the seventh loss value; there are no restrictions on how the seventh loss value is determined.

[0141] For example, the seventh loss value of the i-th sample image can be determined using the following formula (8):

[0142] Formula (8)

[0143] In formula (3), This represents the seventh loss value corresponding to the i-th sample image.

[0144] Step S29: Determine the second target loss value based on the seventh loss value of each sample image.

[0145] For example, after obtaining the seventh loss value for each sample image in the sample image set, the average of the seventh loss values ​​of all sample images can be calculated, and this average value can be used as the second target loss value.

[0146] At this point, step 209 is complete, and the second target loss value is obtained.

[0147] Step 210: Adjust the network parameters of the initial image encoder based on the second target loss value to obtain the trained target image encoder. For example, the network parameters of the text encoder and the target prompt word learning network can be frozen, that is, the network parameters of the text encoder and the target prompt word learning network are not adjusted. Based on the frozen network parameters of the text encoder and the target prompt word learning network, the network parameters of the initial image encoder are adjusted based on the second target loss value.

[0148] For example, when adjusting the network parameters of the initial image encoder based on the second target loss value, the adjustment goal is to make the second target loss value smaller and smaller. For example, the network parameters of the initial image encoder can be adjusted by using gradient descent or other methods. There are no restrictions on this adjustment process.

[0149] After adjusting the network parameters of the initial image encoder based on the second target loss value, the adjusted image encoder can be obtained. If the adjusted image encoder has converged, it is used as the target image encoder, thus obtaining the trained target image encoder. If the adjusted image encoder has not converged, it is used as the initial image encoder. The sample image is input into the initial image encoder to obtain image features and identity features. The sample image is input into the target prompt word learning network to obtain the second prompt word. The second prompt word is input into the text encoder to obtain the second text features. Step 209 is repeated, and so on, until the trained target image encoder is output.

[0150] For example, after obtaining the target image encoder, a second training phase is completed. In this phase, multimodal features (such as text features) are used to assist in optimizing the image features of the image encoder, resulting in better generalization ability. The network parameters (weights) of the image encoder are updated, while the network parameters (weights) of the text encoder remain frozen.

[0151] In this embodiment, to achieve better performance, different loss value determination methods are used in the first and second stages, i.e., loss values ​​are designed specifically for each stage. For example, in the second stage, a fourth loss value is designed for identity features, which maps samples of the same category to similar feature spaces, driving the image encoder to learn features that distinguish different identities (IDs). A fifth loss value is designed for image features, which learns discriminative features, making the distance between samples of the same category smaller than the distance between samples of different categories, enabling hard sample mining and improving the discriminative ability of the image encoder. A sixth loss value is designed for cross-modal features (such as image features, category features, and text features), which aligns feature representations of different modalities, improving the robustness of image features through cross-modal information. Finally, the fourth, fifth, and sixth loss values ​​are weighted to obtain the final loss value, enabling the image encoder to achieve better performance.

[0152] In one possible implementation, after obtaining the target image encoder, it can be applied to the pedestrian re-identification process. Specifically, the target image encoder retrieves whether an object in a first image to be detected is the same as an object in a second image to be detected. For example, the first image to be detected can be input to the target image encoder, which processes it to obtain first image features. Similarly, the second image to be detected can be input to the target image encoder, which processes it to obtain second image features. This allows for the determination of the similarity between the first and second image features (e.g., Euclidean distance, cosine similarity, cosine distance, etc.). If the similarity is greater than a threshold, the object in the first and second images is determined to be the same; otherwise, if the similarity is not greater than the threshold, they are determined to be different.

[0153] For example, a pedestrian re-identification detection model can include a target prompt word learning network, a target image encoder, and a text encoder. The target image encoder can include a first sub-network and a second sub-network. In the process of pedestrian re-identification, the first sub-network of the target image encoder can be used to extract image features, without involving the processing of the target prompt word learning network and the text encoder, and without needing to use the second sub-network of the target image encoder to extract identity features.

[0154] As can be seen from the above technical solutions, in this embodiment, even under complex conditions such as pose, clothing changes and accessories, human perspective, camera perspective, and human occlusion in real-world usage scenarios, the same object can still be accurately identified. Training the target image encoder based on multi-pose clothing change data improves its robustness and accuracy in multi-pose clothing change scenarios, enhances feature extraction capabilities, and helps to better extract features of various attributes such as clothing, pose, age, and gender, resulting in a more reliable pedestrian re-identification model and achieving cross-modal pedestrian re-identification. Regarding data augmentation, a stable duffision diffusion model is used to amplify and generate clothing change data and multi-pose human image data. This results in richer samples, better performance, improved stability and controllability, higher image quality, and support for high-resolution image generation. The settings for generated scenarios and prompts are more targeted, primarily generating clothing change data and human images in different poses. The model's cross-modal description capability is used to assist in the training of the image encoder. Through the model's cross-modal capability, feature information extraction is aided, enhancing the model's feature extraction ability. Combining text features and image features improves the model's robustness. For the generation of text features, prompt words can be designed to help the model better extract various attributes such as clothing, posture, age, and gender, thereby improving the model's robustness and accuracy in multi-posture scenarios of changing clothes, resulting in a more robust pedestrian re-identification model.

[0155] Based on the same concept as the methods described above, this application proposes an image retrieval device, see [link to relevant documentation]. Figure 5 The diagram shown is a structural schematic of the image retrieval device, which may include:

[0156] The acquisition module 51 is used to acquire the original image and text prompt of the target object, wherein the text prompt is used to indicate the target attribute dimension of the original image; adjust the image content in the original image corresponding to the target attribute dimension based on the text prompt to obtain a multi-dimensional image of the target object; construct a sample image set, wherein the sample image set includes multiple sample images of each target object, wherein the multiple sample images include the original image of the target object and the multi-dimensional image of the target object;

[0157] The training module 52 is used to adjust the network parameters of the initial image encoder to be trained based on the sample image set to obtain the trained target image encoder; wherein, the target image encoder is used to retrieve whether the object in the first image to be detected and the object in the second image to be detected are the same object.

[0158] For example, the target attribute dimension includes at least one of clothing attribute dimension, posture attribute dimension, background attribute dimension, and character attribute dimension; when the acquisition module 51 adjusts the image content in the original image corresponding to the target attribute dimension based on the text prompt to obtain a multi-dimensional image of the target object, it is specifically used as follows: if the target attribute dimension includes the clothing attribute dimension, then the image content in the original image corresponding to the clothing attribute is adjusted based on the text prompt to obtain a multi-dimensional image; if the target attribute dimension includes the background attribute dimension, then the image content in the original image corresponding to the background attribute is adjusted based on the text prompt to obtain a multi-dimensional image; if the target attribute dimension includes the character attribute dimension, then the image content in the original image corresponding to the character attribute is adjusted based on the text prompt to obtain a multi-dimensional image; if the target attribute dimension includes the posture attribute dimension, then the image content in the original image corresponding to the posture attribute is adjusted based on the text prompt and the acquired posture skeleton diagram to obtain a multi-dimensional image.

[0159] For example, when the training module 52 adjusts the network parameters of the initial image encoder to be trained based on the sample image set to obtain the trained target image encoder, it specifically performs the following steps: inputting sample images from the sample image set into the initial image encoder to obtain image features and identity features; inputting the sample images into the initial prompt word learning network to be trained to obtain a first prompt word, and inputting the first prompt word into the text encoder to obtain a first text feature; determining a first target loss value based on the image features and the first text feature; adjusting the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network; inputting the sample images into the target prompt word learning network to obtain a second prompt word, and inputting the second prompt word into the text encoder to obtain a second text feature; determining a second target loss value based on the image features, the second text feature, and the identity feature; and adjusting the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder.

[0160] For example, when the training module 52 determines the first target loss value based on the image features and the first text features, it specifically performs the following steps: determining the first loss value corresponding to the i-th sample image based on the fusion features between the image features of the i-th sample image and the first text features of the i-th sample image, and the fusion features between the image features of the i-th sample image and the first text features of each sample image; determining the second loss value corresponding to the i-th sample image based on the fusion features between the image features of the i-th sample image and the first text features of the i-th sample image, and the fusion features between the first text features of the i-th sample image and the image features of each sample image; determining the third loss value of the i-th sample image based on the first loss value and the second loss value; and determining the first target loss value based on the third loss value of each sample image.

[0161] For example, for each sample image in the sample image set, the identity features of the sample image include the predicted probability value of each category supported by the initial image encoder, and the category corresponding to the maximum predicted probability value is taken as the target category of the sample image; when the training module 52 determines the second target loss value based on the image features, the second text features, and the identity features, it is specifically used to: determine a fourth loss value based on the predicted probability value of the i-th sample image under each category; and determine a fifth loss value for the i-th sample image based on a first distance value and a second distance value, where the first distance value is the distance between the image features of the i-th sample image and the image features of the first class of sample images, and the second distance value is the distance between the image features of the i-th sample image and the image features of the second class of sample images. The distance value of the image features of this image; wherein, the target category of the first type of sample image is the same as the target category of the i-th sample image, and the target category of the second type of sample image is different from the target category of the i-th sample image; the category feature of the category is determined based on the second text feature of the sample image corresponding to the category; the sixth loss value is determined based on the fusion feature between the image feature of the i-th sample image and the category feature of each category, and the fusion feature between the image feature of the i-th sample image and the second text feature of each sample image; the seventh loss value of the i-th sample image is determined based on the fourth loss value, the fifth loss value and the sixth loss value, and the second target loss value is determined based on the seventh loss value of each sample image.

[0162] For example, the training module 52 is further configured to determine the category parameter value of the i-th sample image based on the maximum prediction probability value corresponding to the i-th sample image and the maximum prediction probability value corresponding to each sample image; when the training module 52 determines the fourth loss value based on the prediction probability value of the i-th sample image under each category, it is specifically configured to: determine the fourth loss value of the i-th sample image based on the prediction probability value of the i-th sample image under each category and the category parameter value of the i-th sample image; when the training module 52 determines the sixth loss value based on the fusion feature between the image features of the i-th sample image and the category features of each category, and the fusion feature between the image features of the i-th sample image and the second text features of each sample image, it is specifically configured to: determine the sixth loss value based on the fusion feature between the image features of the i-th sample image and the category features of each category, the fusion feature between the image features of the i-th sample image and the second text features of each sample image, and the category parameter value of the i-th sample image.

[0163] For example, the initial prompt word learning network or the target prompt word learning network includes a text template, which includes multiple attribute dimensions. The text template is used to instruct the initial prompt word learning network or the target prompt word learning network to output prompt words of the multiple attribute dimensions. The multiple attribute dimensions include clothing attribute dimensions, posture attribute dimensions, background attribute dimensions, and character attribute dimensions. When the training module 52 adjusts the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network, it specifically involves: freezing the network parameters of the text encoder and the network parameters of the initial image encoder, and adjusting the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network. When the training module 52 adjusts the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder, it specifically involves: freezing the network parameters of the text encoder and the network parameters of the target prompt word learning network, and adjusting the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder.

[0164] Based on the same concept as the above method, this application proposes an electronic device, see [link to previous application]. Figure 6 As shown, the electronic device includes a processor 61 and a machine-readable storage medium 62, the machine-readable storage medium 62 storing machine-executable instructions that can be executed by the processor 61; the processor 61 is used to execute the machine-executable instructions to implement the image retrieval method disclosed in the above example of this application.

[0165] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the image retrieval method disclosed in the above examples of this application.

[0166] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0167] Based on the same concept as the methods described above, this application also provides a computer program product, which may include a computer program. When executed by a processor, the computer program implements the image retrieval method disclosed in the examples above.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image retrieval method, characterized in that, The method includes: The original image and text prompts of the target object are obtained, wherein the text prompts are used to indicate the target attribute dimension of the original image; the image content in the original image corresponding to the target attribute dimension is adjusted based on the text prompts to obtain a multi-dimensional image of the target object; Construct a sample image set, which includes multiple sample images of each target object, including the original image of the target object and a multi-dimensional image of the target object; The network parameters of the initial image encoder to be trained are adjusted based on the sample image set to obtain the trained target image encoder; wherein, the target image encoder is used to retrieve whether the object in the first image to be detected and the object in the second image to be detected are the same object; The process of adjusting the network parameters of the initial image encoder to be trained based on the sample image set to obtain the trained target image encoder specifically includes: The sample images in the sample image set are input into an initial image encoder to obtain image features and identity features; the sample images are input into an initial prompt word learning network to be trained to obtain a first prompt word; the first prompt word is input into a text encoder to obtain a first text feature; a first target loss value is determined based on the image features and the first text feature; the network parameters of the initial prompt word learning network are adjusted based on the first target loss value to obtain a trained target prompt word learning network; The sample image is input into the target prompt word learning network to obtain a second prompt word, and the second prompt word is input into the text encoder to obtain a second text feature; a second target loss value is determined based on the image feature, the second text feature, and the identity feature; the network parameters of the initial image encoder are adjusted based on the second target loss value to obtain the target image encoder; For each sample image in the sample image set, the identity features of the sample image include the predicted probability value of each category supported by the initial image encoder.

2. The method according to claim 1, characterized in that, The target attribute dimension includes at least one of the following: clothing attribute dimension, posture attribute dimension, background attribute dimension, and character attribute dimension; The process of adjusting the image content in the original image corresponding to the target attribute dimension based on the text prompt words to obtain a multi-dimensional image of the target object includes: If the target attribute dimension includes the clothing attribute dimension, then the image content in the original image corresponding to the clothing attribute is adjusted based on the text prompt to obtain a multi-dimensional image; If the target attribute dimension includes the background attribute dimension, then the image content in the original image corresponding to the background attribute is adjusted based on the text prompt to obtain a multi-dimensional image; If the target attribute dimension includes the person attribute dimension, then the image content in the original image corresponding to the person attribute is adjusted based on the text prompt words to obtain a multi-dimensional image; If the target attribute dimension includes the pose attribute dimension, then the image content in the original image corresponding to the pose attribute is adjusted based on the text prompt and the pose skeleton diagram to obtain a multi-dimensional image.

3. The method according to claim 1, characterized in that, Determining the first target loss value based on the image features and the first text features includes: Based on the fusion feature between the image features of the i-th sample image and the first text feature of the i-th sample image in the sample image set, and the fusion feature between the image features of the i-th sample image and the first text feature of each sample image, the first loss value corresponding to the i-th sample image is determined; Based on the fusion features between the image features of the i-th sample image and the first text features of the i-th sample image, and the fusion features between the first text features of the i-th sample image and the image features of each sample image, the second loss value corresponding to the i-th sample image is determined; A third loss value for the i-th sample image is determined based on the first loss value and the second loss value, and the first target loss value is determined based on the third loss value for each sample image.

4. The method according to claim 1, characterized in that, For each sample image in the sample image set, based on the predicted probability value of each category supported by the initial image encoder, the category corresponding to the maximum predicted probability value is taken as the target category of the sample image; Determining the second target loss value based on the image features, the second text features, and the identity features includes: The fourth loss value is determined based on the predicted probability value of the i-th sample image under each category; The fifth loss value of the i-th sample image is determined based on the first distance value and the second distance value. The first distance value is the distance between the image features of the i-th sample image and the image features of the first class of sample images, and the second distance value is the distance between the image features of the i-th sample image and the image features of the second class of sample images. The target category of the first class of sample images is the same as the target category of the i-th sample image, and the target category of the second class of sample images is different from the target category of the i-th sample image. The category features of a category are determined based on the second text features of the sample images corresponding to the category; the sixth loss value is determined based on the fusion features between the image features of the i-th sample image and the category features of each category, and the fusion features between the image features of the i-th sample image and the second text features of each sample image. The seventh loss value of the i-th sample image is determined based on the fourth loss value, the fifth loss value, and the sixth loss value, and the second target loss value is determined based on the seventh loss value of each sample image.

5. The method according to claim 4, characterized in that, Before determining the fourth loss value based on the predicted probability value of the i-th sample image under each category, the method further includes: determining the category parameter value of the i-th sample image based on the maximum predicted probability value corresponding to the i-th sample image and the maximum predicted probability value corresponding to each sample image; The step of determining the fourth loss value based on the predicted probability value of the i-th sample image under each category includes: determining the fourth loss value of the i-th sample image based on the predicted probability value of the i-th sample image under each category and the category parameter value of the i-th sample image; The determination of the sixth loss value based on the fusion features between the image features of the i-th sample image and the category features of each category, and the fusion features between the image features of the i-th sample image and the second text features of each sample image, includes: determining the sixth loss value based on the fusion features between the image features of the i-th sample image and the category features of each category, the fusion features between the image features of the i-th sample image and the second text features of each sample image, and the category parameter value of the i-th sample image.

6. The method according to claim 1, characterized in that, The initial prompt word learning network or the target prompt word learning network includes a text template, which includes multiple attribute dimensions. The text template is used to instruct the initial prompt word learning network or the target prompt word learning network to output prompt words of the multiple attribute dimensions; wherein, the multiple attribute dimensions include clothing attribute dimension, posture attribute dimension, background attribute dimension, and character attribute dimension. The step of adjusting the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network includes: freezing the network parameters of the text encoder and the network parameters of the initial image encoder, and adjusting the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network. The step of adjusting the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder includes: freezing the network parameters of the text encoder and the network parameters of the target prompt word learning network, and adjusting the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder.

7. An image retrieval device, characterized in that, The device includes: An acquisition module is used to acquire the original image and text prompts of the target object, wherein the text prompts are used to indicate the target attribute dimension of the original image; adjust the image content in the original image corresponding to the target attribute dimension based on the text prompts to obtain a multi-dimensional image of the target object; construct a sample image set, wherein the sample image set includes multiple sample images of each target object, wherein the multiple sample images include the original image of the target object and the multi-dimensional image of the target object; The training module is used to adjust the network parameters of the initial image encoder to be trained based on the sample image set to obtain the trained target image encoder; wherein, the target image encoder is used to retrieve whether the object in the first image to be detected and the object in the second image to be detected are the same object; Specifically, the training module adjusts the network parameters of the initial image encoder to be trained based on the sample image set to obtain the trained target image encoder, and is used for: The sample images in the sample image set are input into an initial image encoder to obtain image features and identity features; the sample images are input into an initial prompt word learning network to be trained to obtain a first prompt word; the first prompt word is input into a text encoder to obtain a first text feature; a first target loss value is determined based on the image features and the first text feature; the network parameters of the initial prompt word learning network are adjusted based on the first target loss value to obtain a trained target prompt word learning network; The sample image is input into the target prompt word learning network to obtain a second prompt word, and the second prompt word is input into the text encoder to obtain a second text feature; a second target loss value is determined based on the image feature, the second text feature, and the identity feature; the network parameters of the initial image encoder are adjusted based on the second target loss value to obtain the target image encoder; For each sample image in the sample image set, the identity features of the sample image include the predicted probability value of each category supported by the initial image encoder.

8. The apparatus according to claim 7, characterized in that, in, The target attribute dimension includes at least one of clothing attribute dimension, posture attribute dimension, background attribute dimension, and character attribute dimension. When the acquisition module adjusts the image content in the original image corresponding to the target attribute dimension based on the text prompt to obtain a multi-dimensional image of the target object, it specifically performs the following: If the target attribute dimension includes a clothing attribute dimension, then the image content in the original image corresponding to the clothing attribute is adjusted based on the text prompt to obtain a multi-dimensional image; if the target attribute dimension includes a background attribute dimension, then the image content in the original image corresponding to the background attribute is adjusted based on the text prompt to obtain a multi-dimensional image; if the target attribute dimension includes a character attribute dimension, then the image content in the original image corresponding to the character attribute is adjusted based on the text prompt to obtain a multi-dimensional image; if the target attribute dimension includes a posture attribute dimension, then the image content in the original image corresponding to the posture attribute is adjusted based on the text prompt and the acquired posture skeleton diagram to obtain a multi-dimensional image. Specifically, when the training module determines the first target loss value based on the image features and the first text features, it is used to: determine the first loss value corresponding to the i-th sample image based on the fusion features between the image features of the i-th sample image and the first text features of the i-th sample image, and the fusion features between the image features of the i-th sample image and the first text features of each sample image; determine the second loss value corresponding to the i-th sample image based on the fusion features between the image features of the i-th sample image and the first text features of the i-th sample image, and the fusion features between the first text features of the i-th sample image and the image features of each sample image; determine the third loss value of the i-th sample image based on the first loss value and the second loss value; and determine the first target loss value based on the third loss value of each sample image. Specifically, for each sample image in the sample image set, based on the predicted probability values ​​of each category supported by the initial image encoder, the category corresponding to the maximum predicted probability value is taken as the target category of that sample image. When the training module determines the second target loss value based on the image features, the second text features, and the identity features, it specifically performs the following: determining a fourth loss value based on the predicted probability values ​​of the i-th sample image under each category; and determining a fifth loss value for the i-th sample image based on a first distance value and a second distance value, where the first distance value is the distance between the image features of the i-th sample image and the image features of the first class of sample images, and the second distance value is the distance between the image features of the i-th sample image and the image features of the second class of sample images. The distance value of the feature; wherein, the target category of the first type of sample image is the same as the target category of the i-th sample image, and the target category of the second type of sample image is different from the target category of the i-th sample image; the category feature of the category is determined based on the second text feature of the sample image corresponding to the category; the sixth loss value is determined based on the fusion feature between the image feature of the i-th sample image and the category feature of each category, and the fusion feature between the image feature of the i-th sample image and the second text feature of each sample image; the seventh loss value of the i-th sample image is determined based on the fourth loss value, the fifth loss value and the sixth loss value; and the second target loss value is determined based on the seventh loss value of each sample image. The training module is further configured to determine the category parameter value of the i-th sample image based on the maximum prediction probability value corresponding to the i-th sample image and the maximum prediction probability value corresponding to each sample image; when determining the fourth loss value based on the prediction probability value of the i-th sample image under each category, the training module is specifically configured to: determine the fourth loss value of the i-th sample image based on the prediction probability value of the i-th sample image under each category and the category parameter value of the i-th sample image; when determining the sixth loss value based on the fusion feature between the image features of the i-th sample image and the category features of each category, and the fusion feature between the image features of the i-th sample image and the second text features of each sample image, the training module is specifically configured to: determine the sixth loss value based on the fusion feature between the image features of the i-th sample image and the category features of each category, the fusion feature between the image features of the i-th sample image and the second text features of each sample image, and the category parameter value of the i-th sample image. The initial prompt word learning network or the target prompt word learning network includes a text template, which includes multiple attribute dimensions. The text template is used to instruct the initial prompt word learning network or the target prompt word learning network to output prompt words of the multiple attribute dimensions. The multiple attribute dimensions include clothing attribute dimension, posture attribute dimension, background attribute dimension, and character attribute dimension. The training module adjusts the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network. Specifically, this is done by: freezing the network parameters of the text encoder and the network parameters of the initial image encoder, and adjusting the network parameters of the initial prompt word learning network based on the first target loss value to obtain the trained target prompt word learning network. The training module adjusts the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder. Specifically, this involves: freezing the network parameters of the text encoder and the network parameters of the target prompt word learning network, and adjusting the network parameters of the initial image encoder based on the second target loss value to obtain the target image encoder.

9. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-6.

Images