Text-to-image generation method and device, equipment and medium
By obtaining description text and target reference posture, and using encoding, extraction and fusion strategies to generate target images, the problem of low processing efficiency in existing technologies is solved, and precise control and efficient generation of character postures are achieved.
Patent Information
- Application Number
- CN202510847348.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-23
- Publication Date
- 2025-10-03
AI Technical Summary
Existing text-to-image generation methods have low processing efficiency when generating images of people in specific postures in the medical and financial fields, mainly because traditional methods find it difficult to achieve precise posture control at the key point level and rely on high-cost three-dimensional posture data annotation.
By obtaining the description text and the target reference posture, the encoding strategy is used to extract the text features, the extraction strategy is used to extract the posture features, and the two are fused into fusion features through the fusion strategy. Finally, the generation strategy is used to generate the target image to achieve precise control of the character posture.
The processing efficiency of text-to-image generation is improved, precise control of the character posture in the target image is achieved, and the data acquisition cost is reduced.
Smart Images
Figure CN120747271A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, device, equipment and medium for generating text into an image. Background Art
[0002] Driven by generative adversarial networks (GANs) and diffusion models, text-to-image generation has made significant progress, enabling the generation of high-quality images from natural language descriptions. However, existing methods still face challenges in generating images of people in specific poses in fields such as medicine and finance. The main reasons are: traditional text-conditional generation models rely on implicit encoding of text descriptions to control poses, making it difficult to achieve precise control at the keypoint level; and supervised learning-based pose control methods require large amounts of annotated 3D pose data, which is expensive and has limited generalization capabilities, resulting in low processing efficiency.
[0003] Therefore, existing text-to-image generation methods suffer from low processing efficiency. Summary of the Invention
[0004] Embodiments of the present invention provide a text-to-image generation method, apparatus, device, and medium, aiming to solve the problem of low processing efficiency in the prior art text-to-image generation method.
[0005] In order to solve the above problems, in a first aspect, an embodiment of the present invention provides a method for generating text to an image, which includes:
[0006] Get description text and target reference pose;
[0007] Encoding the description text based on the encoding strategy to obtain text features;
[0008] Extracting the target reference posture based on the extraction strategy to obtain posture features;
[0009] Based on a fusion strategy, the text feature and the posture feature are fused to obtain a fusion feature;
[0010] The fusion features are generated and processed based on the generation strategy to obtain a target image.
[0011] In a second aspect, an embodiment of the present application provides a text-to-image generation device, comprising:
[0012] An acquisition unit, used to acquire description text and target reference posture;
[0013] An encoding unit, configured to encode the description text based on an encoding strategy to obtain text features;
[0014] An extraction unit, configured to extract the target reference posture based on an extraction strategy to obtain a posture feature;
[0015] A fusion unit, configured to fuse the text feature and the posture feature based on a fusion strategy to obtain a fusion feature;
[0016] A generating unit is used to generate the fusion features based on a generating strategy to obtain a target image.
[0017] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor connected to the memory; the memory is used to store a computer program, and the processor is used to run the computer program stored in the memory to execute the method described in the first aspect above.
[0018] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method described in the first aspect is implemented.
[0019] Embodiments of the present invention provide a text-to-image generation method, apparatus, device, and medium. The method comprises: obtaining a description text and a target reference posture; encoding the description text based on an encoding strategy to obtain text features; extracting the target reference posture based on an extraction strategy to obtain posture features; fusing the text features and posture features based on a fusion strategy to obtain fused features; and generating the fused features based on a generation strategy to obtain a target image. Therefore, the embodiments of the present invention achieve precise control of the posture of a person in the target image by encoding, extracting, and fusing the description text and the target reference posture based on an encoding strategy, an extraction strategy, a fusion strategy, and a generation strategy to obtain fused features, and then generating the fused features based on a generation strategy to obtain a target image. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0021] Figure 1 A flowchart of a text-to-image generation method provided by an embodiment of the present invention;
[0022] Figure 2 A schematic block diagram of a text-to-image generation device provided by an embodiment of the present invention;
[0023] Figure 3 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0025] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0026] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.
[0027] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0028] See also Figure 1 , Figure 1 Schematic diagram of the flow of the text-to-image generation method provided by the embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a method for generating text to image, which includes the following steps S110-S150.
[0029] S110: Obtain description text and target reference posture.
[0030] In this embodiment, the descriptive text is a description of the task image for generating a specific posture input by the user through an input box or other means. The specific posture may include sitting posture, standing posture, walking posture, hand gesture posture, facial expression posture and other character postures; the target reference posture is a reference posture determined based on the descriptive text.
[0031] The specific application scenarios of the embodiments of the present invention may be scenarios such as medical health, financial services, or educational services; for example, the embodiments of the present invention may be applied to specific scenarios such as intelligent customer service images and virtual idols in the fields of medical health, financial services, or educational services to enhance user immersion and product differentiation and competitiveness.
[0032] In one embodiment, before obtaining the description text and the target reference posture, the method further includes:
[0033] Enter the description text;
[0034] Among multiple optional reference postures, the input description text is selected and processed based on a text extraction strategy to obtain the target reference posture.
[0035] In this embodiment, after obtaining the description text, the input description text is selected from a plurality of preset selectable reference postures based on a text extraction strategy to obtain the target reference posture. The target reference posture is obtained by selecting the input description text from the plurality of selectable reference postures based on the text extraction strategy, including: extracting the description text using extraction rules to obtain keywords; and comparing the keywords among the plurality of selectable reference postures to obtain the target reference posture. The extraction rules may be extracting the description text using a preset gesture vocabulary; and the gesture vocabulary may be a vocabulary for various gestures such as sitting, standing, walking, hand gestures, and facial expressions.
[0036] Through the above embodiment, it can be seen that the target reference pose is obtained by selecting the input description text from multiple selectable reference poses based on the input description text. Therefore, selecting the input description text from multiple selectable reference poses based on the text extraction strategy to obtain the target reference pose improves processing efficiency and data accuracy, and allows for subsequent targeted processing based on the description text and the target reference pose.
[0037] S120: Encode the description text based on the encoding strategy to obtain text features.
[0038] In this embodiment, after the description text is acquired, the description text is encoded based on an encoding strategy to obtain the text features.
[0039] In one embodiment, encoding the description text based on the encoding strategy to obtain text features includes:
[0040] Inputting the description text into a text encoder;
[0041] The text encoder is used to convert the description text to obtain the text features.
[0042] In this embodiment, the description text is input into the text encoder, and the description text is converted by the text encoder to obtain the text features. Specifically, the input text is subjected to a second encoding process based on the first preset model to obtain the language tag encoding.
[0043] The text encoder may be CLIP (Contrastive Language–Image Pretraining), T5 (Text-to-Text Transfer Transformer), etc. The present invention does not impose any limitation thereto and may be set according to actual conditions.
[0044] Through the above embodiment, it can be seen that the description text is input into the text encoder; the text encoder is used to convert the description text to obtain the text features. Therefore, the description text is encoded based on the encoding strategy to obtain text features, thereby improving processing efficiency.
[0045] S130 : Extracting the target reference posture based on an extraction strategy to obtain posture features.
[0046] In this embodiment, the target reference posture may be extracted based on an extraction strategy to obtain the posture feature.
[0047] In one embodiment, before extracting the target reference posture based on the extraction strategy to obtain the posture feature, the method further includes:
[0048] Obtain video data and initial model;
[0049] Sampling the video data to obtain target frame data;
[0050] The target frame data is input into the initial model, and the initial model is trained using a training strategy and a contrast loss strategy to obtain a target model.
[0051] In this embodiment, before extracting the target reference posture based on the extraction strategy to obtain posture features, the initial model is trained to obtain the target model; video data and the initial model are obtained, wherein the video data can be a video of the posture, and the initial model can be a posture estimator using a VisionTransformer architecture. The sampling process of the video data to obtain the target frame data includes: performing frame extraction on the video data to obtain intermediate frame data, that is, extracting the video frames into pictures, maintaining the frame order between the pictures; and performing random sampling on the intermediate frame data to obtain the target frame data.
[0052] The target frame data is input into the initial model, and the initial model is trained using the training strategy and the contrast loss strategy to obtain the target model; wherein the formula corresponding to the training strategy is h t =MLP(ViT(Conv3D(v t ))), where v t Represents the t-th frame of video data, that is, the target frame data; Conv3D is a three-dimensional convolutional layer, ViT is a visual Transformer encoder, and MLP is a multi-layer perceptron; Symbol description: is the output posture embedding vector; dimension d h =256; Conv3D has a kernel size of 3×3×3, a stride of 1×2×2, and 64 output channels; ViT is a 12-layer Transformer with 8 attention heads per layer and a hidden layer dimension of 768. The contrastive loss strategy is designed to perform self-supervised training to achieve temporal consistency; the formula corresponding to the contrastive loss strategy is
[0053]
[0054] Where s(·,·) is the cosine similarity function; the temperature coefficient τ = 0.1; and Embedding vector for posture; is the jth negative sample in the negative sample pool; batch size N = 32; number of negative samples K = 1024.
[0055] Through the above embodiment, it can be seen that video data and an initial model are acquired; the video data is sampled and processed to obtain target frame data; the target frame data is input into the initial model, and the initial model is trained using a training strategy and a contrastive loss strategy to obtain a target model. Therefore, the self-supervised learning paradigm eliminates the reliance on 3D pose annotated data, enabling the system to utilize massive amounts of unlabeled video data for training, significantly reducing data acquisition costs, thereby improving processing efficiency and facilitating subsequent targeted processing based on the target model.
[0056] In one embodiment, the step of extracting the target reference posture based on the extraction strategy to obtain the posture feature includes:
[0057] Inputting the target reference posture into a target model;
[0058] The target reference posture is inferred based on the target model to obtain the posture feature.
[0059] In this embodiment, the target reference posture is input into the target model; the target reference posture is inferred based on the target model to obtain the posture feature (i.e., the posture embedding vector mentioned above); specifically, in the inference stage, an image (i.e., the target reference posture) can be used for inference, and the dual-branch posture estimator (i.e., the target model) shares weights, which is equivalent to having only one network, and the image can be input into any branch.
[0060] Through the above embodiment, it can be seen that the target reference posture is input into the target model; and the target reference posture is inferred based on the target model to obtain the posture feature. Therefore, inputting the target reference posture into the trained target model to infer the target reference posture to obtain the posture feature improves processing efficiency.
[0061] S140 , fusing the text feature and the posture feature based on a fusion strategy to obtain a fusion feature.
[0062] In this embodiment, the text feature and the gesture feature may be fused based on a fusion strategy to obtain the fused feature.
[0063] In one embodiment, the fusing of the text feature and the gesture feature based on a fusion strategy to obtain a fused feature includes:
[0064] Converting the text feature and the posture feature to obtain an intermediate feature;
[0065] The text feature and the intermediate feature are calculated and processed to obtain the fusion feature.
[0066] In this embodiment, the text features and the posture features are converted to obtain intermediate features, and the text features and the intermediate features are calculated to obtain the fusion features. Specifically, the text features and the posture features are fused using a fusion strategy to obtain the fusion features. The formula corresponding to the fusion strategy can be:
[0067]
[0068] Among them, the text features are The posture characteristics are Q=z t W Q , K=HW K , V=HW V , the projection matrix can be learned Key-value dimension d k =d z =64, LayerNorm is the layer normalization operation.
[0069] Through the above embodiment, it can be seen that the text features and the posture features are converted to obtain intermediate features, and the text features and the intermediate features are calculated to obtain the fused features. Therefore, by fusing the text features and the posture features through a fusion strategy to obtain fused features, fine-grained alignment of text features and posture features is achieved, and key point positioning errors are significantly reduced compared to traditional conditional splicing methods, thereby improving processing efficiency.
[0070] S150: Generate a target image by processing the fusion features based on a generation strategy.
[0071] In one embodiment, the generating process of the fusion features based on the generation strategy to obtain the target image includes:
[0072] Using a generator to generate the fusion features to obtain an intermediate image;
[0073] The intermediate image is optimized using a posture discrimination strategy to obtain the target image.
[0074] In this embodiment, the generator can be a diffusion model or GAN; the posture discrimination strategy is to use the HRNet pre-trained posture estimator Ψ to construct multi-scale constraints, and the corresponding formula can be
[0075]
[0076] Among them, z t ′ is the fusion feature; h ref is the intermediate feature; lis the l-th layer feature of HRNet (l = 1 corresponds to the shallow layer, l = 3 corresponds to the deep layer); λ l are the level weights (1.0, 0.5, 0.2); ||·||1 is the L1 norm.
[0077] Through the above embodiment, it can be seen that the generator generates and processes the fused features to obtain an intermediate image, and the pose discrimination strategy optimizes and processes the intermediate image to obtain the target image. Therefore, generating and processing the fused features based on the generation strategy to obtain the target image achieves precise control of the pose of the person in the target image, improving processing efficiency.
[0078] In summary, the embodiments of the present invention obtain a description text and a target reference posture; encode the description text based on an encoding strategy to obtain text features; extract the target reference posture based on an extraction strategy to obtain posture features; fuse the text features and the posture features based on a fusion strategy to obtain fused features; and generate the fused features based on a generation strategy to obtain a target image. Therefore, the embodiments of the present invention achieve precise control of the posture of the person in the target image and improve processing efficiency by encoding, extracting, and fusing the description text and the target reference posture based on an encoding strategy, an extraction strategy, a fusion strategy, and a generation strategy to obtain fused features, and then generating the fused features based on a generation strategy to obtain a target image.
[0079] Figure 2 Schematic block diagram of a text-to-image generation device provided by an embodiment of the present invention. Figure 2 As shown, the embodiment of the present invention provides a text-to-image generation device 700 for implementing the above method. Figure 2 , the text-to-image generating apparatus 700 includes:
[0080] An acquisition unit 701 is used to acquire description text and target reference posture;
[0081] An encoding unit 702 is configured to encode the description text based on an encoding strategy to obtain text features;
[0082] An extraction unit 703 is configured to extract the target reference posture based on an extraction strategy to obtain a posture feature;
[0083] A fusion unit 704 is configured to fuse the text feature and the posture feature based on a fusion strategy to obtain a fusion feature;
[0084] The generating unit 705 is configured to generate the fusion features based on a generating strategy to obtain a target image.
[0085] In some embodiments, before executing the step of extracting the target reference posture based on the extraction strategy to obtain posture features, the extraction unit 703 is further configured to:
[0086] Obtain video data and initial model;
[0087] Sampling the video data to obtain target frame data;
[0088] The target frame data is input into the initial model, and the initial model is trained using a training strategy and a contrast loss strategy to obtain a target model.
[0089] In some embodiments, when the extraction unit 703 performs the step of extracting the target reference posture based on the extraction strategy to obtain the posture feature, it is specifically configured to:
[0090] Inputting the target reference posture into a target model;
[0091] The target reference posture is inferred based on the target model to obtain the posture feature.
[0092] In some embodiments, when executing the step of fusing the text feature and the gesture feature based on the fusion strategy to obtain a fused feature, the fusion unit 704 is specifically configured to:
[0093] Converting the text feature and the posture feature to obtain an intermediate feature;
[0094] The text feature and the intermediate feature are calculated and processed to obtain the fusion feature.
[0095] In some embodiments, when the encoding unit 702 performs the step of encoding the description text based on the encoding strategy to obtain text features, it is specifically configured to:
[0096] Inputting the description text into a text encoder;
[0097] The text encoder is used to convert the description text to obtain the text features.
[0098] In some embodiments, when executing the step of generating the fusion features based on the generation strategy to obtain the target image, the generation unit 705 is specifically configured to:
[0099] Using a generator to generate the fusion features to obtain an intermediate image;
[0100] The intermediate image is optimized using a posture discrimination strategy to obtain the target image.
[0101] In some embodiments, before executing the step of acquiring the description text and the target reference posture, the acquiring unit 701 is further configured to:
[0102] Enter the description text;
[0103] Among multiple optional reference postures, the input description text is selected and processed based on a text extraction strategy to obtain the target reference posture.
[0104] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned device can refer to the corresponding description in the aforementioned method embodiment. For the convenience and brevity of description, it will not be repeated here.
[0105] The above device can be implemented in the form of a computer program, which can be used in Figure 3 Runs on the computer equipment shown.
[0106] See also Figure 3 , Figure 3 8 is a schematic block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 800 can be a terminal or a server, wherein the terminal can be an electronic device with communication functions. The server can be a standalone server or a server cluster consisting of multiple servers.
[0107] See Figure 3 The electronic device 800 includes a processor 802 , a memory, and a network interface 805 connected via a system bus 801 , wherein the memory may include a non-volatile storage medium 803 and an internal memory 804 .
[0108] The non-volatile storage medium 803 may store an operating system 8031 and a computer program 8032. The computer program 8032 includes program instructions, which, when executed, may enable the processor 802 to perform a text-to-image generation method.
[0109] The processor 802 is used to provide computing and control capabilities to support the operation of the entire electronic device 800.
[0110] The internal memory 804 provides an environment for the operation of the computer program 8032 in the non-volatile storage medium 803. When the computer program 8032 is executed by the processor 802, the processor 802 can execute a text-to-image generation method.
[0111] The network interface 805 is used to communicate with other devices over the network. Figure 3The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device 800 to which the solution of the present invention is applied. The specific electronic device 800 may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.
[0112] The processor 802 is configured to execute a computer program 8032 stored in the memory to implement the following steps:
[0113] Get description text and target reference pose;
[0114] Encoding the description text based on the encoding strategy to obtain text features;
[0115] Extracting the target reference posture based on the extraction strategy to obtain posture features;
[0116] Based on a fusion strategy, the text feature and the posture feature are fused to obtain a fusion feature;
[0117] The fusion features are generated and processed based on the generation strategy to obtain a target image.
[0118] In some embodiments, before implementing the step of extracting the target reference posture based on the extraction strategy to obtain posture features, the processor 802 is further configured to:
[0119] Obtain video data and initial model;
[0120] Sampling the video data to obtain target frame data;
[0121] The target frame data is input into the initial model, and the initial model is trained using a training strategy and a contrast loss strategy to obtain a target model.
[0122] In some embodiments, when the processor 802 implements the step of extracting the target reference posture based on the extraction strategy to obtain the posture feature, it is specifically configured to:
[0123] Inputting the target reference posture into a target model;
[0124] The target reference posture is inferred based on the target model to obtain the posture feature.
[0125] In some embodiments, when the processor 802 implements the step of fusing the text feature and the gesture feature based on the fusion strategy to obtain a fused feature, it is specifically configured to:
[0126] Converting the text feature and the posture feature to obtain an intermediate feature;
[0127] The text feature and the intermediate feature are calculated and processed to obtain the fusion feature.
[0128] In some embodiments, when implementing the step of encoding the description text based on the encoding strategy to obtain text features, the processor 802 is specifically configured to:
[0129] Inputting the description text into a text encoder;
[0130] The text encoder is used to convert the description text to obtain the text features.
[0131] In some embodiments, when implementing the step of generating the fusion features based on the generation strategy to obtain the target image, the processor 802 is specifically configured to:
[0132] Using a generator to generate the fusion features to obtain an intermediate image;
[0133] The intermediate image is optimized using a posture discrimination strategy to obtain the target image.
[0134] In some embodiments, before implementing the processing step of obtaining the description text and the target reference posture, the processor 802 is further configured to:
[0135] Enter the description text;
[0136] Among multiple optional reference postures, the input description text is selected and processed based on a text extraction strategy to obtain the target reference posture.
[0137] It should be understood that in the embodiment of the present invention, the processor 802 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0138] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.
[0139] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor performs the following steps:
[0140] Get description text and target reference pose;
[0141] Encoding the description text based on the encoding strategy to obtain text features;
[0142] Extracting the target reference posture based on the extraction strategy to obtain posture features;
[0143] Based on a fusion strategy, the text feature and the posture feature are fused to obtain a fusion feature;
[0144] The fusion features are generated and processed based on the generation strategy to obtain a target image.
[0145] In one embodiment, before executing the program instructions to implement the step of extracting the target reference posture based on the extraction strategy to obtain posture features, the processor is further configured to:
[0146] Obtain video data and initial model;
[0147] Sampling the video data to obtain target frame data;
[0148] The target frame data is input into the initial model, and the initial model is trained using a training strategy and a contrast loss strategy to obtain a target model.
[0149] In one embodiment, when the processor executes the program instructions to implement the step of extracting the target reference posture based on the extraction strategy to obtain the posture feature, it is specifically configured to:
[0150] Inputting the target reference posture into a target model;
[0151] The target reference posture is inferred based on the target model to obtain the posture feature.
[0152] In one embodiment, when the processor executes the program instructions to implement the step of fusing the text feature and the gesture feature based on the fusion strategy to obtain a fused feature, the processor is specifically configured to:
[0153] Converting the text feature and the posture feature to obtain an intermediate feature;
[0154] The text feature and the intermediate feature are calculated and processed to obtain the fusion feature.
[0155] In one embodiment, when the processor executes the program instructions to implement the step of encoding the description text based on the encoding strategy to obtain text features, it is specifically configured to:
[0156] Inputting the description text into a text encoder;
[0157] The text encoder is used to convert the description text to obtain the text features.
[0158] In one embodiment, when the processor executes the program instructions to implement the step of generating the fusion features based on the generation strategy to obtain the target image, it is specifically configured to:
[0159] Using a generator to generate the fusion features to obtain an intermediate image;
[0160] The intermediate image is optimized using a posture discrimination strategy to obtain the target image.
[0161] In one embodiment, before executing the program instructions to implement the processing step of obtaining the description text and the target reference posture, the processor is further configured to:
[0162] Enter the description text;
[0163] Among multiple optional reference postures, the input description text is selected and processed based on a text extraction strategy to obtain the target reference posture.
[0164] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.
[0165] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0166] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.
[0167] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.
[0168] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing an electronic device (such as a personal computer, terminal, or network device) to execute all or part of the steps of the method described in various embodiments of the present invention.
[0169] The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The software tools, models, or components that appear in the embodiments of the present invention are only examples and do not represent actual use.
Claims
1. A text-to-image generation method, characterized in that: include: Get description text and target reference pose; Encoding the description text based on the encoding strategy to obtain text features; Extracting the target reference posture based on the extraction strategy to obtain posture features; Based on a fusion strategy, the text feature and the posture feature are fused to obtain a fusion feature; The fusion features are generated and processed based on the generation strategy to obtain a target image.
2. The method according to claim 1, characterized in that Before extracting and processing the target reference posture based on the extraction strategy to obtain posture features, the method further includes: Obtain video data and initial model; Sampling the video data to obtain target frame data; The target frame data is input into the initial model, and the initial model is trained using a training strategy and a contrast loss strategy to obtain a target model.
3. The method according to claim 2, characterized in that The step of extracting the target reference posture based on the extraction strategy to obtain posture features includes: inputting the target reference posture into the target model; The target reference posture is inferred based on the target model to obtain the posture feature.
4. The method according to claim 1, wherein The fusing the text feature and the posture feature based on the fusion strategy to obtain the fusion feature includes: Converting the text feature and the posture feature to obtain an intermediate feature; The text feature and the intermediate feature are calculated and processed to obtain the fusion feature.
5. The method according to claim 1, wherein The encoding process of the description text based on the encoding strategy to obtain text features includes: Inputting the description text into a text encoder; The text encoder is used to convert the description text to obtain the text features.
6. The method according to claim 1, characterized in that The generating process of the fusion features based on the generation strategy to obtain the target image includes: Using a generator to generate the fusion features to obtain an intermediate image; The intermediate image is optimized using a posture discrimination strategy to obtain the target image.
7. The method according to claim 1, characterized in that Before obtaining the description text and the target reference posture, the method further includes: Enter the description text; Among multiple optional reference postures, the input description text is selected and processed based on a text extraction strategy to obtain the target reference posture.
8. A text-to-image generation device, characterized in that: include: An acquisition unit, used to acquire description text and target reference posture; An encoding unit, configured to encode the description text based on an encoding strategy to obtain text features; An extraction unit, configured to extract the target reference posture based on an extraction strategy to obtain a posture feature; A fusion unit, configured to fuse the text feature and the posture feature based on a fusion strategy to obtain a fusion feature; A generating unit is used to generate the fusion features based on a generating strategy to obtain a target image.
9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.