Image generation model training method and related equipment

Abstract

The invention provides an image generation model training method and related equipment, and the method comprises the steps: generating a first text prompt and an RGBA semantic graph based on a plurality of obtained pedestrian images, and adding an identity ID tag of a first pedestrian in the first text prompt; and performing iterative training on a preset model based on the constructed training data to obtain an image generation model, the training data including a first sample pair composed of a pedestrian image, a first text prompt and an RGBA semantic graph, and the image generation model being used for generating a multi-view output image of the target object according to the input image of the target object. Pedestrian images under different scenes and view angles have better generalization ability and are more stable and reliable in practical application, and a high-quality target object multi-view angle output image can be generated.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image generation model training method and related equipment. Background Art

[0002] In large-scale surveillance scenarios, person re-identification (re-ID) determines the identity by analyzing the appearance features of the person in multi-view images captured by different cameras. For example, by analyzing the appearance features of the person in multi-view images captured by different cameras, such as clothing, body posture, walking style, etc., the identity of the person in different scenes, time and different cameras can be identified.

[0003] Existing image generation models rely on the quantity and diversity of training data. However, there are many limitations in obtaining a large number of annotated samples required for supervised learning. For example, collecting and annotating data sets consumes manpower and material resources, and the image perspectives are often incomplete due to the influence of monitoring equipment. In addition, the collected red-green-blue (RGB) images may leak personal information. Due to personal privacy concerns, many subjects are unwilling to participate in the test, resulting in a lack of effective training data, making the multi-view images generated by existing image generation models less effective. Summary of the invention

[0004] The embodiment of the present invention provides an image generation model training method and related equipment to solve the problem that the existing image generation model has poor effect in generating multi-view images.

[0005] To solve the above technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides an image generation model training method, the method comprising:

[0007] Generate a first text prompt and a red, green, and blue RGBA semantic map containing transparency based on the acquired multiple pedestrian images, each pedestrian image corresponds to one first text prompt and one RGBA semantic map, and the first text prompt is added with an identity ID tag of the first pedestrian, the same pedestrian has the same ID tag, and the first pedestrian is any pedestrian in the multiple pedestrian images;

[0008] The preset model is iteratively trained based on the constructed training data to obtain an image generation model, wherein the training data includes a first sample pair consisting of the pedestrian image, the first text prompt and the RGBA semantic map, and the image generation model is used to generate a multi-perspective output image of the target object based on the input image of the target object.

[0009] Optionally, the iterative training of a preset model based on the constructed training data to obtain an image generation model includes:

[0010] Iteratively train the preset model according to the constructed training data to obtain an initial generation model;

[0011] Calculating consistency loss according to a first pedestrian image, a second pedestrian image, a third pedestrian image, and a fourth pedestrian image, wherein the first pedestrian image and the second pedestrian image are images corresponding to different ID labels among the plurality of pedestrian images, and the third pedestrian image and the fourth pedestrian image are images of the same pedestrian from different perspectives generated by the initial generation model according to the first pedestrian image;

[0012] When the consistency loss is less than or equal to a preset threshold, the initial generation model is determined as the image generation model.

[0013] Optionally, after iteratively training the preset model based on the constructed training data to obtain the image generation model, the method further includes:

[0014] A second text prompt is generated based on any two of the multiple first text prompts, wherein the second text prompt has an ID tag of a second pedestrian added thereto, the second pedestrian being a pedestrian other than the first pedestrian, and the second text prompt and the multi-perspective output image of the target object are used to constitute training data for a target model, and the target model is a model linked after the image generation model.

[0015] Optionally, generating a second text prompt according to any two of the first text prompts includes:

[0016] Extracting a first feature tensor of a first prompt and a second feature tensor of a second prompt, wherein the first prompt and the second prompt are any two text prompts with different ID tags among the plurality of first text prompts;

[0017] The second text prompt is generated according to a third feature tensor, where the third feature tensor is obtained by weighted fusion of the first feature tensor and the second feature tensor.

[0018] Optionally, the generating of a first text prompt and a red, green, and blue color RGBA semantic map including transparency based on the acquired multiple pedestrian images includes:

[0019] Inputting the pedestrian image into a pre-trained image-to-text model to obtain a text description corresponding to the pedestrian image;

[0020] Adding an ID tag to the text description to obtain the first text prompt; and,

[0021] Input the pedestrian image into a pre-trained 3D mesh generator to obtain a 3D mesh corresponding to the pedestrian image;

[0022] The 3D mesh is projected to obtain the RGBA semantic map, wherein the A channel in the RGBA semantic map contains distance information from the vertices of the 3D mesh to the viewpoint, and the distance information is used to describe the posture information of the pedestrian.

[0023] Optionally, the consistency loss is calculated using the following formula:

[0024] L all =L Control +β(L vc +L′ vc +L′ v ' c );

[0025] Among them, L all is the consistency loss, L control is the mean square error loss of the initial generative model, β is a predefined margin value, and L vc is the loss between the first pedestrian image and the third pedestrian image, L' vc is the loss between the first pedestrian image and the fourth pedestrian image, L” vc is the loss between the third pedestrian image and the fourth pedestrian image.

[0026] In a second aspect, an embodiment of the present invention provides an image generation model training device, the device comprising:

[0027] A first generating module is used to generate a first text prompt and a red, green and blue RGBA semantic map containing transparency based on the acquired multiple pedestrian images, each pedestrian image corresponds to one first text prompt and one RGBA semantic map, and the first text prompt is added with an identity ID tag of the first pedestrian, the same pedestrian has the same ID tag, and the first pedestrian is any pedestrian in the multiple pedestrian images;

[0028] A training module is used to iteratively train a preset model based on constructed training data to obtain an image generation model, wherein the training data includes a first sample pair consisting of the pedestrian image, the first text prompt and the RGBA semantic map, and the image generation model is used to generate a multi-perspective output image of the target object based on an input image of the target object.

[0029] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0030] at least one processor; and

[0031] a memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to the first aspect.

[0033] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0034] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0035] In the embodiment of the present invention, by generating a first text prompt and an RGBA semantic map for each pedestrian image, and adding the ID tag of the first pedestrian in the first text prompt, the original pedestrian image data is enriched. The first text prompt describes various features of the pedestrian in text form, providing more semantic information for the model, and the ID tag is added to the first text prompt, which realizes the accurate identification of the pedestrian identity, helps the model learn the common features of the same pedestrian under different perspectives, and improves the accuracy and reliability of pedestrian re-identification; the RGBA semantic map adds transparency and depth semantics on the basis of color information, better reflects the pedestrian posture, and provides a more comprehensive and accurate information basis for subsequent model training and image generation. The preset model is iteratively trained based on the training data of the first sample pair consisting of the pedestrian image, the first text prompt and the RGBA semantic map, so that the model can fully learn the complex relationship between the pedestrian image, the text prompt and the RGBA semantic map. It has better generalization ability for pedestrian images under different scenes and perspectives, is more stable and reliable in practical applications, and can generate high-quality multi-perspective output images of target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 It is one of the flow charts of the image generation model training method provided by an embodiment of the present invention;

[0038] Figure 2 This is one of the schematic diagrams for comparing multi-view image generation provided by an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of generating a multi-view image of a new ID provided by an embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of the overall framework of the image generation model training method provided by an embodiment of the present invention;

[0041] Figure 5 This is the second schematic diagram of multi-view image generation comparison provided by an embodiment of the present invention;

[0042] Figure 6 This is the second flowchart of the image generation model training method provided by an embodiment of the present invention;

[0043] Figure 7 It is a structural schematic diagram of an image generation model training device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1 , Figure 1 is one of the flow charts of the image generation model training method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0046] Step 101: Generate a first text prompt and a red, green, and blue RGBA semantic map containing transparency based on the acquired multiple pedestrian images, each pedestrian image corresponds to one first text prompt and one RGBA semantic map, and the first text prompt is added with an identity ID tag of a first pedestrian, the same pedestrian has the same ID tag, and the first pedestrian is any pedestrian in the multiple pedestrian images;

[0047] In this step, multiple original images are first obtained, which are the basic data for subsequent operations. Specifically, the original images can be images captured from video clips shot by surveillance cameras, public pedestrian datasets, or images collected specifically for training and testing pedestrian re-identification models, etc., in order to obtain sufficiently rich and diverse pedestrian data. In some examples, the original images may contain complex backgrounds, other irrelevant objects or people, and cropping can be used to focus on the pedestrian subject and remove unnecessary interference information, so that the cropped image presents the appearance characteristics of the pedestrian, which is convenient for subsequent classification and recognition processing.

[0048] Then, the cropped images are classified according to the different identity features of the pedestrians. For example, the images can be classified into different categories according to the obvious identity features of the pedestrians, such as clothing style, gender, age group, etc., to ensure that images of the same category have similarities in certain identity features and that different categories are distinguishable, laying the foundation for subsequent accurate identity recognition and model training.

[0049] After that, we can refer to the annotation method of the Market-1501 dataset and assign a unique identity (ID) label to each classified pedestrian image. The ID label can be used to clearly identify the identity of the pedestrian in the image. When training the model, the model can establish the correspondence between image features and identity labels by learning these labeled images, thereby realizing the identity recognition of new unlabeled pedestrian images. It helps to learn the common features of the same pedestrian under different perspectives, so that when generating multi-perspective output images of the target object, it can better maintain identity consistency, avoid confusing the features of different pedestrians, and improve the accuracy and reliability of pedestrian re-identification. For example, for a pedestrian image of a certain clothing style and gender, a specific number or string is assigned as its identity label so that different pedestrian identities can be accurately identified and distinguished in subsequent data processing and model training. In this way, multiple pedestrian images are obtained after cropping, classifying and assigning labels to multiple original images.

[0050] Optionally, the step 101 of generating a first text prompt and a red, green, and blue color RGBA semantic map including transparency based on the acquired multiple pedestrian images includes:

[0051] Inputting the pedestrian image into a pre-trained image-to-text model to obtain a text description corresponding to the pedestrian image;

[0052] An ID tag is added to the text description to obtain the first text prompt.

[0053] In this example, a pre-trained image-to-text model can be used to generate corresponding text descriptions for each pedestrian image. The pre-trained image-to-text model can be BLIP2, which has been pre-trained on large-scale image-text pair data and has the ability to extract key information from the image and generate corresponding text descriptions. Before inputting each pedestrian image into the BLIP2 model, some pre-processing operations can be performed on the image, such as adjusting the image size to the input size expected by the model, ensuring that the image format meets the model requirements (for example, uniformly adjusting to RGB format), and normalizing the image so that the pixel value of the image is within a suitable range, so that the model can better process the image data.

[0054] The BLIP2 model first extracts features from the input pedestrian image. Using deep learning techniques such as convolutional neural networks, it identifies various visual elements in the image, such as the color, style, texture of the pedestrian's clothing, body posture (standing, walking, running, etc.), and other significant appearance features (hairstyle, accessories, etc.). Based on the extracted image features, the BLIP2 model generates corresponding text descriptions through its internal language generation mechanism according to the pre-learned association pattern between image features and text descriptions. For example, if the pedestrian in the image is wearing a red dress, the model may generate a text description such as "a lady wearing a red dress" to describe the appearance features of the pedestrian in the image as accurately as possible. In addition, some post-processing operations may be required for the generated text description. For example, check the grammatical correctness and fluency of the text, correct some possibly inaccurate or vague descriptions, or format the text according to specific application requirements to make it more in line with the requirements of subsequent operations (such as adding ID tags, etc.).

[0055] After obtaining the text description corresponding to the pedestrian image, an ID tag is added to the text description to obtain the first text prompt. Specifically, a series of rare tags that have not been touched by stable diffusion can be selected. These rare tags can be some unique and uncommon words or phrases, such as some uncommon color names, descriptions of clothing in a specific style (such as "Byzantine-style cloak"), unique accessory names (such as "emerald-encrusted hairpin"), etc., to ensure that these tags are unique and distinguishable in subsequent processing, and can accurately refer to the pedestrian ID in the image. For each pedestrian image, according to its unique appearance features or other relevant attributes, a suitable rare tag is selected from the determined rare tag set to be associated with the pedestrian's ID. For example, if a pedestrian in a pedestrian image wears a national costume with a unique pattern, then a rare tag related to the style of the national costume can be selected as part of its ID tag.

[0056] Then, the selected rare label is added to the text description previously generated by the BLIP2 model. The location of the addition can be determined according to the specific needs and the structure of the text description. For example, it can be placed after the key part describing the appearance characteristics of the pedestrian, or at the beginning or end of the entire text description to highlight the association between the label and the pedestrian ID. Exemplarily, if the text description is "a man wearing blue jeans and a white T-shirt", and the selected rare label is "emerald emerald necklace", then the first text prompt after adding the ID label can be "a man wearing blue jeans and a white T-shirt and wearing an emerald emerald necklace", where "emerald emerald necklace" is the rare label part that specifically refers to the pedestrian ID. In this way, the rare label with the pedestrian ID information is integrated into the text description, forming the first text prompt containing the ID label, so that the identity information of different pedestrians can be accurately identified and distinguished in the subsequent image generation and processing process.

[0057] Optionally, the step 101 of generating a first text prompt and a red, green, and blue color RGBA semantic map including transparency based on the acquired multiple pedestrian images includes:

[0058] Input the pedestrian image into a pre-trained 3D mesh generator to obtain a 3D mesh corresponding to the pedestrian image;

[0059] The 3D mesh is projected to obtain the RGBA semantic map, wherein the A channel in the RGBA semantic map contains distance information from the vertices of the 3D mesh to the viewpoint, and the distance information is used to describe the posture information of the pedestrian.

[0060] In this example, for each pedestrian image, a pre-trained 3D mesh generator is used to generate a 3D mesh for it. Then the generated 3D mesh is projected onto a transparent red-green-blue-alpha (RGBA) map to obtain the corresponding RGBA semantic map. This semantic map not only contains red, green, and blue color information, but also contains the distance information from the 3D mesh vertex to the viewpoint through the A channel (transparency channel), which can provide more human posture information, such as the front and back distribution of the limbs, the tilt of the trunk and head, and other pedestrian posture information, providing richer semantic information for subsequent image generation and processing.

[0061] Specifically, the pre-trained 3D mesh generator has been learned on a large amount of 3D model data and has the ability to generate corresponding 3D structures based on two-dimensional images. The pedestrian image is input into the 3D mesh generator. The generator first analyzes the input pedestrian image to identify the pedestrian's outline, the proportion of each body part, and the characteristic information of clothing in the image. Through its internal deep learning algorithm, an initial 3D mesh model is constructed based on the mapping relationship between the pre-learned image features and the 3D structure. This model can restore the shape of the pedestrian in three-dimensional space, including the three-dimensional structure of body parts such as the head, torso, and limbs. Then the generated 3D mesh is projected to obtain an RGBA semantic map. During the projection process, the three-dimensional information of the 3D mesh is converted into two-dimensional image information, in which the R (red), G (green), and B (blue) channels respectively store information related to color, similar to the color representation of ordinary images. Among them, the A channel contains the distance information from the vertex of the 3D mesh to the viewpoint (observation point). During the projection process, the distance from each vertex to the viewpoint is calculated and encoded into the A channel. For example, vertices closer to the viewpoint may correspond to higher pixel values ​​(such as white or close to white) in the A channel, while vertices farther from the viewpoint may correspond to lower pixel values ​​(such as black or close to black). This distance information is very important for describing the posture information of pedestrians. For example, when the limbs of a pedestrian are stretched out, the vertices of the limbs far away from the torso will show different distance values ​​from the torso vertices in the projected A channel, so that the position and posture of the pedestrian's limbs, such as standing, walking, bending, etc., can be judged by analyzing the changes in the pixel values ​​of the A channel. Similarly, the inclination of the torso and head can also be reflected in the A channel through the changes in the distances from the vertices of different parts to the viewpoint, providing a rich posture information foundation for subsequent operations such as pedestrian multi-view alignment and identity fusion image generation.

[0062] In this way, through step 101, the first text prompt is generated to supplement the semantic description information for the pedestrian image. Through the pre-trained BLIP2, it is possible to extract and convert the pedestrian from the image into a text form that can be understood by humans, making the content of the image clearer. For example, the characteristics of the pedestrian's clothing, gender, posture, etc. in the image are described in words, which helps to better understand and utilize the information in the image in subsequent processing, not just limited to the pixel data of the image. The RGBA semantic map adds transparency (A channel) on the basis of traditional RGB color information, which can be used to better handle occlusion relationships during operations such as image fusion; and the distance information describes the pedestrian's posture, such as the front and back distribution of the limbs, the inclination of the trunk and the head, etc., so that the image can reflect more three-dimensional posture features on the two-dimensional plane, providing a more accurate posture basis for pedestrian multi-view alignment and identity fusion. In addition, the identity ID tag of the first pedestrian is added to the first text prompt, and the same pedestrian has the same tag, which establishes a unique identity for the pedestrian in the image. In subsequent processing, whether it is the images of the same pedestrian from different perspectives or the images of different pedestrians, they can be accurately distinguished and associated through the ID tag. For example, when training the model, the model can learn the common features of images of the same pedestrian from different perspectives, as well as the difference features between different pedestrians based on the ID labels, thereby improving the accuracy and generalization ability of the pedestrian re-identification model.

[0063] Step 102: iteratively train the preset model based on the constructed training data to obtain an image generation model, wherein the training data includes a first sample pair consisting of the pedestrian image, the first text prompt, and the RGBA semantic map, and the image generation model is used to generate a multi-perspective output image of the target object based on the input image of the target object.

[0064] In this step, the preset model is iteratively trained based on the constructed training data. The preset model can be based on the ControlNet model. The iterative training of the preset model can be fine-tuning the Stable Diffusion v1.5 in the ControlNet model based on the constructed training data, and using the fine-tuned model to generate pedestrian images. As the number of iterations increases, the model gradually learns the complex relationship between pedestrian images and text prompts, RGBA semantic maps, and how to generate high-quality, consistent multi-view output images. Finally, a fully trained image generation model is obtained, which can accurately generate its multi-view output images based on the input image of the target object, such as Figure 2 As shown, the image generation model obtained by the image generation model training method provided by the present invention can provide an effective solution for the pedestrian multi-perspective alignment and identity fusion image generation tasks, and improves the quality of the multi-perspective images compared to the multi-perspective images generated by the image generation model in the prior art.

[0065] In the embodiment of the present invention, by generating a first text prompt and an RGBA semantic map for each pedestrian image, and adding the ID tag of the first pedestrian in the first text prompt, the original pedestrian image data is enriched. The first text prompt describes various features of the pedestrian in text form, providing more semantic information for the model, and the ID tag is added to the first text prompt, which realizes the accurate identification of the pedestrian identity, helps the model learn the common features of the same pedestrian under different perspectives, and improves the accuracy and reliability of pedestrian re-identification; the RGBA semantic map adds transparency and depth semantics on the basis of color information, better reflects the pedestrian posture, and provides a more comprehensive and accurate information basis for subsequent model training and image generation. The preset model is iteratively trained based on the training data of the first sample pair consisting of the pedestrian image, the first text prompt and the RGBA semantic map, so that the model can fully learn the complex relationship between the pedestrian image, the text prompt and the RGBA semantic map. It has better generalization ability for pedestrian images under different scenes and perspectives, is more stable and reliable in practical applications, and can generate high-quality multi-perspective output images of target objects.

[0066] Optionally, after the step 102, iteratively training the preset model based on the constructed training data to obtain the image generation model, the method further includes:

[0067] A second text prompt is generated based on any two of the multiple first text prompts, wherein the second text prompt has an ID tag of a second pedestrian added thereto, the second pedestrian being a pedestrian other than the first pedestrian, and the second text prompt and the multi-perspective output image of the target object are used to constitute training data for a target model, and the target model is a model linked after the image generation model.

[0068] In this embodiment, any two different first text prompts are randomly selected from multiple first text prompts. For these two text prompts, the pedestrian appearance feature description, semantic information and other contents contained therein are extracted, and then the information is fused. The fusion method can adopt weighted average, splicing and other methods. For example, for the part describing the pedestrian's clothing, the clothing description in the two text prompts can be combined according to a certain weight. At the same time, an ID tag of a second pedestrian (i.e., a pedestrian other than the first pedestrian) is added to it. This ID tag can be selected from a pre-prepared unused ID tag set, or a new ID tag is generated according to a specific rule to ensure its uniqueness. In this way, a second text prompt containing new ID information and fused features is generated. By generating a second text prompt, new pedestrian identity information is introduced to solve the problem of limited diversity of existing IDs. In the task of pedestrian re-identification, data diversity is crucial to the generalization ability of the model. More diverse ID tags and feature descriptions enable the model to learn a wider range of pedestrian feature patterns, thereby improving the recognition and generation capabilities of different pedestrians and avoiding performance bottlenecks when the model faces new and unseen pedestrian IDs.

[0069] The generated second text prompt is combined with the multi-view output image of the target object generated by the image generation model to form the training data of the target model. This further expands the scale and diversity of the target model training data and enriches the learning resources of the target model. Based on these training data containing new ID information and diverse features, the target model can learn more about the relationship between multi-view images of pedestrians and text prompts, as well as how to better handle the image generation task of pedestrians with different IDs.

[0070] Optionally, generating a second text prompt according to any two of the first text prompts includes:

[0071] Extracting a first feature tensor of a first prompt and a second feature tensor of a second prompt, wherein the first prompt and the second prompt are any two text prompts with different ID tags among the plurality of first text prompts;

[0072] The second text prompt is generated according to a third feature tensor, where the third feature tensor is obtained by weighted fusion of the first feature tensor and the second feature tensor.

[0073] In this embodiment, considering that the diversity of existing IDs is limited, only data enhancement of existing IDs is prone to bottlenecks. Based on this, generating new IDs can effectively improve diversity and further expand the data set. For details, see the following description:

[0074] For any two first prompts and second prompts with different ID tags in the first text prompt, they can first be input into a pre-trained text encoder (such as Clip text encoder), and the Clip text encoder will convert the text prompt into a corresponding feature tensor representation. Specifically, after the first prompt passes through the text encoder, a first feature tensor is obtained. The first feature tensor can be a mathematical representation of the appearance features, semantic information, etc. of the pedestrian contained in the first prompt in a high-dimensional vector space, which can capture various feature patterns related to the pedestrian. Similarly, the second prompt passes through the same text encoder to obtain a second feature tensor, which reflects the feature information of another different pedestrian.

[0075] After extracting the first feature tensor of the first hint and the second feature tensor of the second hint, determine the weighted fusion coefficient, which can be set based on experience or experiment. Then, the first feature tensor and the second feature tensor are weighted fused according to the set weighted coefficient. For example, determine that the weighted fusion coefficient η between the first feature tensor and the second feature tensor is 0.1, and set any two features with different ID labels as [V i ] and [V j ], then [V i ] and [V j ] The first feature tensor and the second feature tensor extracted by the Clip text encoder are F[V i ] is F[V j ], the third eigentensor can be obtained by the following calculation formula:

[0076] f v =ηF[V i ]+(1-η)F[V j ];

[0077] f v is the third feature tensor. In this way, through this weighted fusion method, the feature information of two different pedestrians is fused together to form a new feature combination, which represents a new pedestrian feature pattern and provides a basis for generating pedestrian images with new IDs.

[0078] Then, based on the generated third feature tensor, a pre-trained language model (such as a related component based on a stable diffusion model) is used to convert the third feature tensor back into text form, thereby generating a second text prompt. This second text prompt contains the fused new feature information, and a new ID label (corresponding to the pedestrian with the new ID) is added. During the generation process, the language model generates a text description that can describe the characteristics of the new pedestrian based on the feature pattern in the third feature tensor, and combines it with the new ID label to form a complete second text prompt. In this way, by extracting the feature tensors of the first text prompts with different ID labels and weightedly fusing them to generate the third feature tensor, and then generating the second text prompt, the purpose of obtaining new ID-related information without additional training in the subsequent reasoning stage is achieved, which provides key text information support for expanding the data set of subsequent target models and generating pedestrian images with new IDs. Figure 3 As shown, Figure 3 These are multi-view images of some of the generated new IDs. It can be seen that the generated new IDs have well integrated the apparent features of the existing IDs.

[0079] Considering that only relying on ID tags to generate images is prone to insufficient constraints. Based on this, Figure 4 As shown, we can use the image as the starting point, the input image and the generated multi-view image as anchors and positive samples, and the real images with different IDs as negative samples. We can use the pre-trained osnet model to extract their features and calculate the triplet loss. For details, please refer to the following statement:

[0080] Optionally, the step 102, iteratively training a preset model based on the constructed training data to obtain an image generation model, includes:

[0081] Iteratively train the preset model according to the constructed training data to obtain an initial generation model;

[0082] Calculating consistency loss according to a first pedestrian image, a second pedestrian image, a third pedestrian image, and a fourth pedestrian image, wherein the first pedestrian image and the second pedestrian image are images corresponding to different ID labels among the plurality of pedestrian images, and the third pedestrian image and the fourth pedestrian image are images of the same pedestrian from different perspectives generated by the initial generation model according to the first pedestrian image;

[0083] When the consistency loss is less than or equal to a preset threshold, the initial generation model is determined as the image generation model.

[0084] In this embodiment, first, the preset model is iteratively trained based on the constructed training data to obtain an initial generation model. During the training process, the model learns how to generate corresponding pedestrian images based on the input image, text prompts and semantic graph information. Then, prepare data for calculating the consistency loss, select the first pedestrian image and the second pedestrian image (they have different ID labels and represent different pedestrians), and the third pedestrian image and the fourth pedestrian image of the same pedestrian from different perspectives generated by the initial generation model based on the first pedestrian image. The first pedestrian image, the second pedestrian image, the third pedestrian image and the fourth pedestrian image will be used as the basic data for calculating the perspective consistency loss.

[0085] The first pedestrian image and the second pedestrian image are used as anchors and negative samples (because they are images of different pedestrians), and the third pedestrian image and the fourth pedestrian image are used as positive samples (because they are images of the same pedestrian from different perspectives). Use a pre-trained re-ID model (such as osnet) to extract the features of these images. Compare the calculated consistency loss with the preset threshold. If the consistency loss is less than or equal to the preset threshold, it means that the initial generation model has good perspective consistency when generating images of the same pedestrian from different perspectives, and can generate images of different perspectives that are relatively consistent in features, meeting the requirements. At this point, the initial generation model is determined as the final image generation model. The model performs well in solving the perspective consistency loss problem and can be used for subsequent pedestrian multi-perspective alignment and identity fusion image generation tasks to generate high-quality pedestrian images with consistent perspectives. If the consistency loss is greater than the preset threshold, it is necessary to further adjust the model parameters, optimize the training process, or improve the model structure, etc., to improve the perspective consistency of the model and reduce the consistency loss until the threshold requirements are met and a suitable image generation model is obtained.

[0086] Optionally, the consistency loss is calculated using the following formula:

[0087] L all =L Control +β(L vc +L′ vc +L′ v ' c );

[0088] Among them, L all is the consistency loss, L control is the mean square error loss of the initial generative model, β is a predefined margin value, and L vc is the loss between the first pedestrian image and the third pedestrian image, L' vc is the loss between the first pedestrian image and the fourth pedestrian image, L” vc is the loss between the third pedestrian image and the fourth pedestrian image.

[0089]

[0090] Among them, E i and F i All are plotted points, P i is a positive sample, N i is a negative sample, f(.) is a feature extraction function, the pre-trained osnet is used here, and α is a pre-defined margin value. The perspective consistency loss can not only constrain the perspective consistency, but also narrow the similarity between the generated image features and the real image features. In order to speed up the convergence of the model and reduce the impact of poor generation quality in the initial stage on the perspective consistency loss, a two-stage training process is adopted. In the first stage, only the mean square error loss function of the diffusion model is used:

[0091]

[0092] where Z0 is the input data, t is uniformly sampled from [1,...,T], ∈ is the noise sampled from a Gaussian distribution, and c t Indicates text prompt, c f Represents an RGBA semantic map. By making the model’s prediction value ∈ θ (z t ,t,c t ,c f ) is as close to the real noise ∈ as possible, and the model parameters θ are optimized to minimize the prediction error.

[0093] After the first stage of training, the perspective consistency loss is added for the second stage of training:

[0094] L all =L Control +β(L vc +L′ vc +L″ vc ).

[0095] In this way, by introducing consistency loss as an important constraint in the model training process, the model is guided to adjust parameters in the direction of generating images with consistent perspectives during training. This helps to optimize the training process and make the model converge to a better state faster. Compared with training without consistency loss constraints, the model can more effectively utilize the information in the training data, avoid falling into local optimal solutions, improve training efficiency, reduce training time and resource consumption, and obtain better model performance. The multi-view images generated by the model optimized by calculating the consistency loss have better consistency, which makes it more accurate and reliable to identify the same pedestrian from different camera perspectives. The model can better learn the features of pedestrians that remain unchanged at different perspectives, reduce recognition errors caused by changes in perspective, and improve the overall accuracy and reliability of the pedestrian re-identification system. Figure 5 As shown, it can be seen that the new perspective image generated by the present invention performs well in terms of authenticity and consistency.

[0096] In some embodiments, Figure 6 As shown, the image generation model training method may include the following steps:

[0097] S11, image acquisition and annotation: collect multiple pedestrian images, manually crop, classify and assign labels (taking Market-1501 as an example).

[0098] S12, construct text description: Generate corresponding text descriptions for the images collected in S11 through the pre-trained image-to-text model BLIP2. Since the processed images are generally single-person images, the generated text descriptions are generally in the format of "a man / woman..."

[0099] S13, construct a text description with an ID tag: add rare tags not touched by StableDiffusion to the text description obtained in S12 to specifically refer to the pedestrian ID in the image, which is used as the ID tag.

[0100] S14, construct RGBA semantic map: Use the pre-trained 3D mesh generator Cliff on the image obtained in S11 to obtain the corresponding 3D mesh, and project these 3D meshes into RGBA semantic maps containing depth information, where the A channel contains the distance information from the vertex of the 3D mesh to the viewpoint, which provides more human posture information such as the front and back distribution of the limbs, the tilt of the torso and head, etc.

[0101] S15, construct a training set: the pedestrian image in S11, the text prompt in S13 and the RGBA semantic map in S14 are combined to form a pedestrian image-RGBA semantic map-text prompt sample pair.

[0102] S2, perspective consistency loss: Take the image as the starting point, the input image and the generated multi-view image as anchors and positive samples, and the real images of different IDs as negative samples. Use the pre-trained osnet model to extract their features and calculate the triplet loss.

[0103] S3, ID fusion: extract text embedding features of different ID tags respectively and weight them as the identity features of the new ID to guide the denoising diffusion model to generate a new ID.

[0104] S4, fine-tuning model: Based on the ControlNet model, S2 and S3 are added to form a fine-tuning model. The fine-tuning model and the training set obtained by S1 are used to fine-tune Stable Diffusion v1.5, and the fine-tuned model is used to generate pedestrian images.

[0105] Finally, generate multi-view images: use the fine-tuned model for image generation.

[0106] In this embodiment, the original pedestrian image data is enriched by generating a first text prompt and an RGBA semantic map for each pedestrian image, and adding the ID tag of the first pedestrian in the first text prompt. The first text prompt describes various features of the pedestrian in text form, providing more semantic information for the model, and adding the ID tag in the first text prompt realizes the accurate identification of the pedestrian identity, which helps the model learn the common features of the same pedestrian under different perspectives and improves the accuracy and reliability of pedestrian re-identification; the RGBA semantic map adds transparency and depth semantics on the basis of color information, better reflects the pedestrian posture, and provides a more comprehensive and accurate information basis for subsequent model training and image generation. The preset model is iteratively trained based on the training data of the first sample pair consisting of the pedestrian image, the first text prompt and the RGBA semantic map, so that the model can fully learn the complex relationship between the pedestrian image, the text prompt and the RGBA semantic map. It has better generalization ability for pedestrian images in different scenes and perspectives, is more stable and reliable in practical applications, and can generate high-quality multi-perspective output images of target objects.

[0107] See also Figure 7 , Figure 7 is a schematic diagram of the structure of the image generation model training device provided by an embodiment of the present invention, such as Figure 7 As shown, the image generation model training device 700 includes:

[0108] The first generating module 701 is used to generate a first text prompt and a red, green and blue RGBA semantic map with transparency based on the acquired multiple pedestrian images, each pedestrian image corresponds to one first text prompt and one RGBA semantic map, and the first text prompt is added with an identity ID tag of the first pedestrian, the same pedestrian has the same ID tag, and the first pedestrian is any pedestrian in the multiple pedestrian images;

[0109] The training module 702 is used to iteratively train the preset model based on the constructed training data to obtain an image generation model, wherein the training data includes a first sample pair consisting of the pedestrian image, the first text prompt and the RGBA semantic map, and the image generation model is used to generate a multi-perspective output image of the target object based on the input image of the target object.

[0110] Optionally, the training module 702 is specifically configured to:

[0111] Iteratively train the preset model according to the constructed training data to obtain an initial generation model;

[0112] Calculating consistency loss according to a first pedestrian image, a second pedestrian image, a third pedestrian image, and a fourth pedestrian image, wherein the first pedestrian image and the second pedestrian image are images corresponding to different ID labels among the plurality of pedestrian images, and the third pedestrian image and the fourth pedestrian image are images of the same pedestrian from different perspectives generated by the initial generation model according to the first pedestrian image;

[0113] When the consistency loss is less than or equal to a preset threshold, the initial generation model is determined as the image generation model.

[0114] Optionally, the device further comprises:

[0115] The second generation module is used to generate a second text prompt based on any two of the multiple first text prompts, the second text prompt is added with an ID tag of a second pedestrian, the second pedestrian is a pedestrian other than the first pedestrian, the second text prompt and the multi-perspective output image of the target object are used to constitute training data for a target model, and the target model is a model linked after the image generation model.

[0116] Optionally, the second generating module is specifically used to:

[0117] Extracting a first feature tensor of a first prompt and a second feature tensor of a second prompt, wherein the first prompt and the second prompt are any two text prompts with different ID tags among the plurality of first text prompts;

[0118] The second text prompt is generated according to a third feature tensor, where the third feature tensor is obtained by weighted fusion of the first feature tensor and the second feature tensor.

[0119] Optionally, the first generating module 701 is specifically configured to:

[0120] Inputting the pedestrian image into a pre-trained image-to-text model to obtain a text description corresponding to the pedestrian image;

[0121] An ID tag is added to the text description to obtain the first text prompt.

[0122] Optionally, the first generating module 701 is specifically configured to:

[0123] Input the pedestrian image into a pre-trained 3D mesh generator to obtain a 3D mesh corresponding to the pedestrian image;

[0124] The 3D mesh is projected to obtain the RGBA semantic map, wherein the A channel in the RGBA semantic map contains distance information from the vertices of the 3D mesh to the viewpoint, and the distance information is used to describe the posture information of the pedestrian.

[0125] Optionally, the consistency loss is calculated using the following formula:

[0126] L all =L Control +β(L vc +L′ vc +L′ v ' c );

[0127] Among them, L all is the consistency loss, L control is the mean square error loss of the initial generative model, β is a predefined margin value, and L vc is the loss between the first pedestrian image and the third pedestrian image, L' vc is the loss between the first pedestrian image and the fourth pedestrian image, L” vc is the loss between the third pedestrian image and the fourth pedestrian image.

[0128] The image generation model training device 700 is capable of implementing each process of each embodiment of the above-mentioned image generation model training method. The technical features correspond one to one and can achieve the same technical effect. To avoid repetition, they will not be described here.

[0129] An embodiment of the present invention further provides an electronic device, including:

[0130] at least one processor; and

[0131] a memory communicatively connected to the at least one processor; wherein,

[0132] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the various processes of the above-mentioned image generation model training method embodiment and achieve the same technical effect. To avoid repetition, they are not repeated here.

[0133] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned image generation model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0134] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned image generation model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0135] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0137] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for training an image generation model, characterized in that: The method comprises: Generate a first text prompt and a red, green, and blue RGBA semantic map containing transparency based on the acquired multiple pedestrian images, each pedestrian image corresponds to one first text prompt and one RGBA semantic map, and the first text prompt is added with an identity ID tag of the first pedestrian, the same pedestrian has the same ID tag, and the first pedestrian is any pedestrian in the multiple pedestrian images; The preset model is iteratively trained based on the constructed training data to obtain an image generation model, wherein the training data includes a first sample pair consisting of the pedestrian image, the first text prompt and the RGBA semantic map, and the image generation model is used to generate a multi-perspective output image of the target object based on the input image of the target object.

2. The method according to claim 1, characterized in that The preset model is iteratively trained based on the constructed training data to obtain an image generation model, including: Iteratively train the preset model according to the constructed training data to obtain an initial generation model; Calculating consistency loss according to a first pedestrian image, a second pedestrian image, a third pedestrian image, and a fourth pedestrian image, wherein the first pedestrian image and the second pedestrian image are images corresponding to different ID labels among the plurality of pedestrian images, and the third pedestrian image and the fourth pedestrian image are images of the same pedestrian from different perspectives generated by the initial generation model according to the first pedestrian image; When the consistency loss is less than or equal to a preset threshold, the initial generation model is determined as the image generation model.

3. The method according to claim 1, characterized in that After iteratively training the preset model based on the constructed training data to obtain the image generation model, the method further includes: A second text prompt is generated based on any two of the multiple first text prompts, wherein the second text prompt has an ID tag of a second pedestrian added thereto, the second pedestrian being a pedestrian other than the first pedestrian, and the second text prompt and the multi-perspective output image of the target object are used to constitute training data for a target model, and the target model is a model linked after the image generation model.

4. The method according to claim 3, characterized in that The step of generating a second text prompt according to any two of the first text prompts includes: Extracting a first feature tensor of a first prompt and a second feature tensor of a second prompt, wherein the first prompt and the second prompt are any two text prompts with different ID tags among the plurality of first text prompts; The second text prompt is generated according to a third feature tensor, where the third feature tensor is obtained by weighted fusion of the first feature tensor and the second feature tensor.

5. The method according to claim 1, characterized in that The method of generating a first text prompt and a red, green and blue color RGBA semantic map containing transparency based on the acquired multiple pedestrian images includes: Inputting the pedestrian image into a pre-trained image-to-text model to obtain a text description corresponding to the pedestrian image; Adding an ID tag to the text description to obtain the first text prompt; and, Input the pedestrian image into a pre-trained 3D mesh generator to obtain a 3D mesh corresponding to the pedestrian image; The 3D mesh is projected to obtain the RGBA semantic map, wherein the A channel in the RGBA semantic map contains distance information from the vertices of the 3D mesh to the viewpoint, and the distance information is used to describe the posture information of the pedestrian.

6. The method according to claim 2, characterized in that The consistency loss is calculated using the following formula: L all =L Control +β(L vc +L′ vc +L′ v ′ c ); Among them, L all is the consistency loss, L control is the mean square error loss of the initial generative model, β is a predefined margin value, and L vc is the loss between the first pedestrian image and the third pedestrian image, L' vc is the loss between the first pedestrian image and the fourth pedestrian image, L” vc is the loss between the third pedestrian image and the fourth pedestrian image.

7. An image generation model training device, characterized in that: The device comprises: A first generating module is used to generate a first text prompt and a red, green and blue RGBA semantic map containing transparency based on the acquired multiple pedestrian images, each pedestrian image corresponds to one first text prompt and one RGBA semantic map, and the first text prompt is added with an identity ID tag of the first pedestrian, the same pedestrian has the same ID tag, and the first pedestrian is any pedestrian in the multiple pedestrian images; A training module is used to iteratively train a preset model based on constructed training data to obtain an image generation model, wherein the training data includes a first sample pair consisting of the pedestrian image, the first text prompt and the RGBA semantic map, and the image generation model is used to generate a multi-perspective output image of the target object based on an input image of the target object.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor so as to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images