Model training method, image generation method, equipment and vehicle

Abstract

The invention relates to the field of data processing, and provides a model training method, an image generation method, equipment and a vehicle, and the method comprises the steps: obtaining a training data set and an initial image generation model, and the training data set comprises image data and text prompt information corresponding to the image data; determining feature information of at least one piece of target data in the picture data; performing feature extraction on the text prompt information and the feature information of the at least one piece of target data to obtain embedded features corresponding to the picture data; and training the initial image generation model by using the picture data and the embedded features until a first preset training ending condition is met, and obtaining an image generation model. After the text prompt information is received subsequently, the image data corresponding to the text prompt information can be generated by using the image generation model, and the generated image data is more fit with the text prompt information, so that the accuracy of the generated image data is improved, and the requirements of a user are met.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a model training method, an image generation method, a device and a vehicle. Background Art

[0002] The model can be used to directly generate high-fidelity images by inputting text. It has been widely used in wallpaper generation, advertising production, auxiliary design and other fields, and has great practical value.

[0003] Although the model has made great achievements, the generated images corresponding to the input text are often inaccurate and cannot meet user needs. Summary of the invention

[0004] In view of this, the purpose of the present disclosure is to propose a model training method, image generation method, device and vehicle to solve the problem that the current generation of images corresponding to the input text based on the input text is often inaccurate and cannot meet user needs.

[0005] Based on the above purpose, the first aspect of the present disclosure provides a model training method, the method comprising:

[0006] Acquire a training data set and an initial image generation model, wherein the training data set includes image data and text prompt information corresponding to the image data;

[0007] Determine feature information of at least one target data in the image data;

[0008] Extracting features from the text prompt information and the feature information of the at least one target data to obtain embedded features corresponding to the image data;

[0009] The initial image generation model is trained using the picture data and the embedded features until a first preset training end condition is met, thereby obtaining an image generation model.

[0010] Based on the same inventive concept, the second aspect of the present disclosure proposes an image generation method, the method comprising:

[0011] Obtaining text prompt information to be processed, and inputting the text prompt information to be processed into an image generation model;

[0012] Identifying the target data to be processed contained in the text prompt information to be processed, and determining the target position information corresponding to the target data to be processed by using the image generation model;

[0013] The image generation model is used to generate and output a target image corresponding to the text prompt information to be processed according to the target position information.

[0014] Based on the same inventive concept, the third aspect of the present disclosure proposes a model training device, the device comprising:

[0015] A data acquisition module is configured to acquire a training data set and an initial image generation model, wherein the training data set includes image data and text prompt information corresponding to the image data;

[0016] a position determination module, configured to determine feature information of at least one target data in the image data;

[0017] A feature extraction module is configured to extract features from the text prompt information and feature information of the at least one target data to obtain embedded features corresponding to the image data;

[0018] The training module is configured to train the initial image generation model using the image data and the embedded features until a first preset training end condition is met to obtain an image generation model.

[0019] Based on the same inventive concept, the fourth aspect of the present disclosure provides an image generating device, the device comprising:

[0020] A data acquisition module is configured to acquire text prompt information to be processed and input the text prompt information to be processed into an image generation model;

[0021] a position determination module, configured to identify the target data to be processed contained in the text prompt information to be processed, and determine the target position information corresponding to the target data to be processed by using the image generation model;

[0022] The image generation module is configured to generate and output a target image corresponding to the text prompt information to be processed according to the target position information by using the image generation model.

[0023] Based on the same inventive concept, the fifth aspect of the present disclosure proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.

[0024] Based on the same inventive concept, the sixth aspect of the present disclosure proposes a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method as described above.

[0025] Based on the same inventive concept, the seventh aspect of the present disclosure provides a vehicle, including the model training device described in the third aspect or the image generating device described in the fourth aspect or the electronic device described in the fifth aspect or the storage medium described in the sixth aspect.

[0026] As can be seen from the above, the present disclosure proposes a model training method, an image generation method, a device and a vehicle, obtains a training data set and an initial image generation model, wherein the training data set includes picture data and text prompt information corresponding to the picture data, so that the initial image generation model can be trained using the training data set later. Determine the feature information of at least one target data in the picture data, extract the feature information of the text prompt information and the feature information of the at least one target data, and obtain the embedded feature corresponding to the picture data. By extracting the feature, when the initial image generation model is subsequently trained according to the training data set, the specific features of the target data are considered, so that the output of the subsequent image generation model contains the target data corresponding to the text prompt information and its corresponding feature information, and the feature information is the same as the feature contained in the text prompt information, that is, after the feature extraction, the specific features of each target data can be determined, and then the output image contains the target data with specific features, so that the output image is more corresponding to the text prompt information, and the output image is more accurate. The initial image generation model is trained using the picture data and the embedded features until the first preset training end condition is met to obtain the image generation model. Subsequently, after receiving the text prompt information, the image generation model can be used to generate image data corresponding to the text prompt information. The generated image data is more consistent with the text prompt information, which improves the accuracy of the generated image data and meets the needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flowchart of a method for training a model according to an embodiment of the present disclosure;

[0029] Figure 2 is a flowchart of an image generating method according to an embodiment of the present disclosure;

[0030] Figure 3 A structural block diagram of a training device for a model according to an embodiment of the present disclosure;

[0031] Figure 4is a structural block diagram of an image generating device according to an embodiment of the present disclosure;

[0032] Figure 5 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0034] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0035] The terms used in this disclosure are explained as follows:

[0036] Prompt: It refers to the text or instructions that provide input to the model to guide it to produce specific output.

[0037] CLIP: The CLIP (Contrastive Language-Image Pre-Training, CLIP) model, whose core concept is image-text contrast learning pre-training, is a multimodal learning model that aims to associate images and texts. It can quickly implement tasks such as image-text feature similarity calculation, cross-modal retrieval, and zero-shot image classification.

[0038] T5: The T5 (Transfer Text-to-Text Transformer, T5) model is a text-to-text converter proposed by Google. It uses the Transformer structure and self-attention mechanism to perform transfer learning for a variety of NLP tasks.

[0039] Embedding model: Embedding model, Embedding is a technology that converts high-dimensional sparse one-hot encoding into low-dimensional dense vector representation, which can be used for dimensionality reduction, dimensionality increase, similarity calculation, etc.

[0040] DQN: Deep Q network (DQN) is an algorithm used to solve reinforcement learning problems.

[0041] DDPG: DDPG (Deep Deterministic Policy Gradient) is an off-policy algorithm and can only be used in continuous action space.

[0042] PPO: Proximal Policy Optimization (PPO) algorithm.

[0043] TRPO: Trust region policy optimization (TRPO).

[0044] A3C: Reinforcement learning algorithm (Asynchronous Advantage Actor-Critic, A3C).

[0045] SAC: SAC is a maximum entropy-based policy learning method (Soft Actor-Critic, SAC), which improves learning speed and stability by adding entropy terms and randomization strategies.

[0046] The model can be used to directly generate high-fidelity images by inputting text. It has been widely used in wallpaper generation, advertising production, auxiliary design and other fields, and has great practical value.

[0047] Although the model has made great achievements, its generation effect is still unsatisfactory in the scenario of generating pictures corresponding to the relative positions contained in the text based on the input text.

[0048] The current diffusion models are mainly LayoutDiffusion, TextDiffuser and TextDiffuser-2. Specifically, LayoutDiffusion can generate target objects of specific categories at specific locations according to the specified layout, but a layout diagram is required. TextDiffuser can first train a "text-layout" model, whose input is text and output is the layout information of the text that needs to be displayed in the image, and then use the text and layout information as conditions to train a diffusion model, so that images containing text can be accurately generated. TextDiffuser-2 goes a step further, first training a Decoder-Only LLM model similar to GPT, whose input is text and output is directly the layout information of the text that needs to be displayed in the image (upper left and lower right corner coordinates), then directly combining the text and coordinates into a new text prompt, and finally training the diffusion model of Text2Image according to conventional ideas. TextDiffuser and TextDiffuser-2 are only for obtaining better text image generation effects, and do not involve the generation of arbitrary target objects. The images generated by the model cannot meet the needs of users.

[0049] Based on the above description, this embodiment proposes a model training method, such as Figure 1 As shown, the method includes:

[0050] Step 101: Acquire a training data set and an initial image generation model, wherein the training data set includes image data and text prompt information corresponding to the image data.

[0051] During the specific implementation, a training data set and an initial image generation model are obtained, wherein the initial parameter image generation model is a model using a neural network structure, including at least one of the following: a diffusion model, a DQN model, a DDPG model, a PPO model, a TRPO model, an A3C model, a SAC model, and in this embodiment, a diffusion model is preferably used.

[0052] The training data set includes picture data and text prompt information corresponding to the picture data, wherein the text prompt information is a text prompt. In this embodiment, the text prompt information corresponding to the picture data means that the text prompt information is text information describing the picture data.

[0053] For example, if the picture data is a picture of a white cat lying on a table, the text prompt information corresponding to the picture data is a white cat on a table, which can also be expressed as “A white cat on the table”.

[0054] Step 102: Determine feature information of at least one target data in the image data.

[0055] In a specific implementation, the image data includes at least one target data, and the target data may be a physical object existing in the image data. By identifying the image data, the at least one target data included in the image data is identified.

[0056] In this embodiment, the characteristic information of the target data specifically includes the location information and category information of the target data, wherein the location information indicates the specific location of the target data in the image data, and the category information indicates the specific category corresponding to the target data.

[0057] Determine the location information of each target data. The specific process may be:

[0058] The image data is mapped to a preset coordinate system, and coordinate information corresponding to each target data contained in the image data is determined, where the coordinate information is the position information corresponding to the target data.

[0059] For example, the image data is a picture of a white cat lying on a table. The target data is identified to obtain the target data contained in the image data, which are the cat and the table. The image data is mapped to a preset coordinate system to obtain the position information of the cat as [x, 0.55], [y, 0.4], and the position information of the table as [x, 0.15], [y, 0.3].

[0060] It is understandable that the determination method of the location information of at least one target data in the image data can also be performed in a designated manner, that is, the user sets the location information of the target data by himself. This embodiment does not limit the specific determination method of the location information of the target data.

[0061] Step 103: extract features from the text prompt information and the feature information of the at least one target data to obtain embedded features corresponding to the image data.

[0062] During specific implementation, feature extraction is performed on the feature information of the text prompt information and the feature information of the at least one target data respectively to obtain embedded features corresponding to the image data.

[0063] As for the feature extraction method, the feature extraction may be performed by using a feature extraction model or by using a preset feature extraction algorithm, which is not specifically limited in this embodiment.

[0064] Step 104: train the initial image generation model using the image data and the embedded features until a first preset training end condition is met to obtain an image generation model.

[0065] During specific implementation, the initial image generation model is trained using the picture data and the embedded features until a first preset training end condition is met to obtain an image generation model.

[0066] The first preset training end condition includes at least one of the following: determining that all new training data are input into the initial image generation model for training, determining that the loss function of the initial image generation model converges to a first preset convergence threshold, or determining that the initial image generation model is iteratively trained to a first preset number of iterations.

[0067] Exemplarily, the first preset training end condition is to determine that all new training data are input into the initial image generation model for training:

[0068] There are ten groups of data in the new training data, each group of data includes image data and embedded features. The first preset training end condition is to determine that all data in the new training data are input into the initial image generation model for training, that is, when the ten groups of data are input into the initial image generation model, there is no data in the new training data that has not been input into the initial image generation model, and the initial image generation model training is determined to be completed, and a new initial image generation model is obtained.

[0069] In another example, the first preset training end condition is to determine that the loss function of the initial image generation model converges to a first preset convergence threshold:

[0070] Input new training data into the initial image generation model for training, and output a first training result. Determine a first loss function based on the first training result and the actual output torque. The type of the first loss function includes at least one of the following: mean square error loss function, cross entropy loss function, logarithmic loss function, exponential loss function, square loss function or absolute value loss function, etc. When the first loss function converges to a first preset convergence threshold, it is determined that the first preset training end condition is met, and a new initial image generation model is obtained.

[0071] In another example, the first preset training end condition is to determine that the initial image generation model is iteratively trained to a first preset number of iterations.

[0072] The new training data is input into the initial image generation model for iterative training, the number of iterations is recorded, and when it is determined that the number of iterations is equal to the first preset number of iterations, the first preset training end condition is met, and a new initial image generation model is obtained.

[0073] In this embodiment, the first preset training end condition is preferably to determine that the loss function of the initial image generation model converges to a first preset convergence threshold.

[0074] Through the above scheme, a training data set and an initial image generation model are obtained, wherein the training data set includes picture data and text prompt information corresponding to the picture data, so that the initial image generation model can be trained using the training data set later. The feature information of at least one target data in the picture data is determined, and the feature information of the text prompt information and the at least one target data is extracted to obtain the embedded features corresponding to the picture data. By extracting the features, when the initial image generation model is subsequently trained according to the training data set, the position information of the target data is considered, so that the output of the subsequent image generation model contains the target data corresponding to the text prompt information and its corresponding feature information, and the feature information is the same as the feature contained in the text prompt information, that is, after the feature extraction, the specific features of each target data can be determined, and then the output image contains the target data with specific features, so that the output image is more corresponding to the text prompt information, and the output image is more accurate. The initial image generation model is trained using the picture data and the embedded features until the first preset training end condition is met, and the image generation model is obtained. Subsequently, after receiving the text prompt information, the image generation model can be used to generate image data corresponding to the text prompt information. The generated image data is more consistent with the text prompt information, which improves the accuracy of the generated image data and meets the needs of users.

[0075] Based on the execution process of step 101 to step 104, the initial image generation model includes a text feature extraction model and an embedding model. In the process of extracting features from the text prompt information and the feature information of the at least one target data to obtain the embedded features corresponding to the image data, the text feature extraction model and the embedding model can be used to perform feature extraction operations. Specifically, in some embodiments, the initial image generation model includes a text feature extraction model and an embedding model, and step 103 specifically includes:

[0076] Step 1031: input the text prompt information into a text feature extraction model, and process it through the text feature extraction model to obtain text embedding features corresponding to the image data.

[0077] Step 1032: input feature information of the at least one target data into an embedding model, and obtain target embedding features corresponding to the image data through processing by the embedding model.

[0078] Step 1033, concatenating the text embedding feature and the target embedding feature to obtain the embedding feature corresponding to the image data.

[0079] During specific implementation, the initial image generation model includes a text feature extraction model and an embedding model. In this embodiment, the embedding model is an embedding model.

[0080] The text prompt information is input into a text feature extraction model, and processed by the text feature extraction model to obtain text embedding features corresponding to the image data. In this embodiment, the text feature extraction model can adopt a CLIP model or a T5 model, which is not specifically limited.

[0081] The feature information of the at least one target data is input into the embedding model, and the embedding model is used to process the target embedding features corresponding to the image data. The text embedding features and the target embedding features are concatenated to obtain the embedding features corresponding to the image data.

[0082] Specifically, the splicing process may be to add the text embedding features and the target embedding features, or to splice by superimposing the number of channels.

[0083] Exemplarily, the text embedding feature and the target embedding feature are added together, specifically, the text embedding feature and the target embedding feature are added together element by element to obtain the embedding feature, so as to promote the flow of gradients, improve training stability, and reduce the gradient vanishing problem in deep networks.

[0084] In another example, the splicing is performed by stacking the number of channels, specifically connecting the text embedding features and the target embedding features together in a certain dimension to generate an embedding feature. The number of channels or feature dimensions can be increased to better capture the relationship between different features in subsequent layers.

[0085] Through the above scheme, by inputting text prompt information and location information into their respective corresponding models, text embedding features and target embedding features are obtained respectively, and the obtained text embedding features and target embedding features are more accurate.

[0086] Based on the execution process of step 103, the text feature extraction model includes multiple initial text feature extraction models, and then in the process of determining the text embedding feature by using the text feature extraction model, multiple initial text feature extraction models can be used to extract features to obtain the text embedding feature. Therefore, in some embodiments, step 1031 specifically includes:

[0087] Step 10311, input the text prompt information into multiple initial text feature extraction models respectively, and process them through the multiple text feature extraction models to obtain multiple initial text embedding features.

[0088] Step 10312, concatenate all the initial text embedding features to obtain the text embedding features corresponding to the image data.

[0089] In a specific implementation, the text feature extraction model includes multiple initial text feature extraction models, and the text prompt information is respectively input into the multiple initial text feature extraction models, and processed by the multiple text feature extraction models to obtain multiple initial text embedding features. Wherein, each initial text feature extraction model is a CLIP model or a T5 model.

[0090] All initial text embedding features are concatenated. In this embodiment, the concatenation is performed in the last dimension to obtain the text embedding features corresponding to the image data.

[0091] For example, the concatenated text embedding feature dimension is d text , the text length is L, then the text embedding feature is represented as [B,L,d text ], where B is the amount of image data.

[0092] Through the above scheme, compared with using a single initial text feature extraction model, multiple initial text feature extraction models are used to improve the expressiveness of the text.

[0093] In the process of inputting the feature information of the at least one target data into the embedding model and processing the embedding model to obtain the target embedding feature corresponding to the image data, since the embedding model includes the first embedding model and the second embedding model, in some embodiments, step 1032 specifically includes:

[0094] Step 10321, classify all target data to obtain category information corresponding to each target data in all target data;

[0095] Step 10322, determining the location information of at least one target data in the image data;

[0096] Step 10323, inputting the category information corresponding to the target data into the first embedding model, and processing the first embedding model to obtain the category embedding features corresponding to the target data;

[0097] Step 10324, inputting the position information corresponding to the target data into a second embedding model, and processing the second embedding model to obtain a coordinate embedding feature corresponding to the target data;

[0098] Step 10325, summing the category embedding feature and the coordinate embedding feature to obtain an initial target embedding feature corresponding to the target data;

[0099] Step 10326, counting the initial target embedding features corresponding to all target data to obtain the target embedding features corresponding to the image data.

[0100] In specific implementation, all target data are classified to obtain category information corresponding to each target data in all target data. In this embodiment, the classification processing method can be that multiple categories are pre-stored in the database, and each target data is compared with multiple categories to obtain category information corresponding to each target data.

[0101] Specifically, the target data in the image data (such as a picture of a white cat lying on a table) is identified to obtain the target data (including the cat and the table) contained in the image data. The target data is compared with each of the multiple categories to determine the category information corresponding to the target data (i.e., the category corresponding to the cat is animals, and the category corresponding to the table is static objects, i.e., the cat is one category and the table is another category).

[0102] Determine the location information of each target data. The specific process may be:

[0103] The image data is mapped to a preset coordinate system, and coordinate information corresponding to each target data contained in the image data is determined, where the coordinate information is the position information corresponding to the target data.

[0104] For example, the image data is a picture of a white cat lying on a table. The target data is identified to obtain the target data contained in the image data, which are the cat and the table. The image data is mapped to a preset coordinate system to obtain the position information of the cat as [x, 0.55], [y, 0.4], and the position information of the table as [x, 0.15], [y, 0.3].

[0105] It is understandable that the determination method of the location information of at least one target data in the image data can also be performed in a designated manner, that is, the user sets the location information of the target data by himself. This embodiment does not limit the specific determination method of the location information of the target data.

[0106] The category information corresponding to the target data is input into the first embedding model, and is processed by the first embedding model to obtain the category embedding features corresponding to the target data.

[0107] For example, if the number of target data corresponding to the category information is n class , the feature dimension of the first embedding model is d class , the target number of target data contained in each image data is k, then the category embedding feature is expressed as [k,4,d class], if the number of target data corresponding to the category information is less than k, it is filled with 0.

[0108] The position information corresponding to the target data is input into a second embedding model, and processed by the second embedding model to obtain a coordinate embedding feature corresponding to the target data.

[0109] For example, if the maximum resolution of the image data is n size , the feature dimension of the second embedding model is d size , the target resolution corresponding to each image data is k, then the coordinate embedding feature is expressed as [k,1,d size ], if the maximum resolution of the image data is less than k, it is padded with 0.

[0110] That is, in this embodiment, each target data corresponds to a category embedding feature and a coordinate embedding feature, and the category embedding feature and the coordinate embedding feature are added together to obtain the initial target embedding feature corresponding to the target data. class =d size After summing, the initial target embedding feature is [k,4,d class ]. At the same time, the data padded with 0 does not participate in the subsequent model training.

[0111] Arrange the initial target embedding features corresponding to all the target data to obtain the target embedding features corresponding to the image data, which can be expressed as [1,4k,d class ]. If the number of image data is B, the target embedding feature corresponding to all image data is expressed as [B,4k,d class ].

[0112] Through the above scheme, different embedding models are used to extract embedded features for category information and position information, and the obtained category embedding features and coordinate embedding features are determined to be more accurate, thereby improving the accuracy of subsequently generated image data.

[0113] Based on the process of executing step 1031 to step 1033, in some embodiments, step 1033 specifically includes:

[0114] Step 10331, projecting the text embedding feature to a preset dimension to obtain the text embedding feature to be processed.

[0115] Step 10332, projecting the target embedding feature to a preset dimension to obtain the target embedding feature to be processed.

[0116] Step 10333, performing word segmentation processing on the text prompt information to obtain the number of word segments.

[0117] Step 10334, determining the target dimension according to the number of word segments, concatenating the embedding features of the text to be processed and the embedding features of the target to be processed in the target dimension to obtain the embedding features corresponding to the image data.

[0118] In a specific implementation, the text embedding feature is projected to a preset dimension to obtain the text embedding feature to be processed. The target embedding feature is projected to a preset dimension to obtain the target embedding feature to be processed.

[0119] The text prompt information is segmented to obtain the number of segmented words. The target dimension is determined according to the number of segmented words, that is, the value corresponding to the number of segmented words is the same as the value corresponding to the target dimension.

[0120] For example, the text prompt information is "A white cat on the table", and the text prompt information is segmented to determine that the number of segmented words is 6, and then the target dimension is determined to be 6.

[0121] The embedding features of the text to be processed and the embedding features of the target to be processed are concatenated in the target dimension to obtain the embedding features corresponding to the image data.

[0122] For example, the text embedding feature is projected to dimension d, and the text embedding feature to be processed is [B, L, d text ], project the target embedding feature to dimension d, and obtain the target embedding feature to be processed as [B,4k,d class ]. After concatenation in the target dimension, the embedding feature corresponding to the image data is obtained as [B,L+4k,d class ].

[0123] Through the above scheme, the embedded features corresponding to the image data include both text embedded features and target embedded features, and the target embedded features include category embedded features and coordinate embedded features. That is, the text, category and coordinates are comprehensively considered when determining the embedded features corresponding to the image data. Therefore, the obtained embedded features include both text and category related information and coordinate information. The obtained embedded features are more comprehensive and specific, and then the embedded features are used to train the initial image generation model in the subsequent training, and the training is more accurate.

[0124] In some embodiments, the process of determining the training data set includes:

[0125] Step A, obtaining training picture data, inputting the training picture data into an initial language model for training, until a second preset training end condition is met, and obtaining a language model.

[0126] Step B: input the image data into the language model, and obtain text prompt information corresponding to the image data through processing by the language model.

[0127] Step C: Aggregate the image data and the text prompt information corresponding to the image data to obtain a training data set.

[0128] In a specific implementation, training picture data is obtained, and the initial language model is trained using the training picture data until a second preset training end condition is met to obtain a language model. The initial language model is a model using a neural network structure.

[0129] The second preset training end condition includes at least one of the following: determining that all new training data are input into the initial language model for training, determining that the loss function of the initial language model converges to a second preset convergence threshold, or determining that the initial language model is iteratively trained to a second preset number of iterations.

[0130] Exemplarily, the second preset training end condition is to determine that all new training data are input into the initial language model for training:

[0131] There are ten groups of data in the new training data, each group of data includes training picture data and text prompt information. The second preset training end condition is to determine that all data in the new training data are input into the initial language model for training, that is, when the ten groups of data are input into the initial language model, there is no data in the new training data that has not been input into the initial language model, and the initial language model training is determined to be completed, and a new initial language model is obtained.

[0132] In another example, the second preset training end condition is to determine that the loss function of the initial language model converges to a second preset convergence threshold:

[0133] Input new training data into the initial language model for training, and output a second training result. Determine a second loss function based on the second training result and the actual output torque. The type of the second loss function includes at least one of the following: mean square error loss function, cross entropy loss function, logarithmic loss function, exponential loss function, square loss function or absolute value loss function, etc. When the second loss function converges to a second preset convergence threshold, it is determined that the second preset training end condition is met, and a new initial language model is obtained.

[0134] In another example, the second preset training end condition is to determine that the initial language model is iteratively trained to a second preset number of iterations.

[0135] The new training data is input into the initial language model for iterative training, the number of iterations is recorded, and when it is determined that the number of iterations is equal to a second preset number of iterations, the second preset training end condition is met, and a new initial language model is obtained.

[0136] The image data is input into the trained language model, and the language model is used to process the image data to obtain text prompt information corresponding to the image data. The image data and the text prompt information corresponding to the image data are aggregated to obtain a training data set required for training the initial image generation model.

[0137] In this embodiment, the format of the training data set is "caption <eos>[class_name][top_left_x][top_left_y][bottom_right_x][bottom_right_y] <eos>…”, where caption is the text prompt information, [class_name] indicates the name of the target data contained in the image data, and [top_left_x][top_left_y][bottom_right_x][bottom_right_y] indicates the location information of the target data.

[0138] Through the above scheme, the text prompt information corresponding to the image data is determined by using the trained language model, which reduces the steps of human participation and improves the speed of determining the training data set while ensuring the accuracy of the training data set.

[0139] In some embodiments, the initial language model includes a multimodal model and a target detection model, and step A specifically includes:

[0140] Step A01, input the training picture data into the multimodal model, and output the training text prompt information and training position information via the multimodal model.

[0141] Step A02: Use the target detection model to extract the position of the training image data to obtain actual position information.

[0142] Step A03, determining a target loss function according to the training position information and the actual position information.

[0143] Step A04, in response to the target loss function converging to a preset convergence threshold, determining that a second preset training end condition is met, and obtaining a language model.

[0144] In a specific implementation, the initial language model includes a multimodal model and a target detection model. The training image data is input into the multimodal model, and the multimodal model outputs training text prompt information and training position information, wherein the training text prompt information is text information describing the image data, and the training position information is position information corresponding to the target data contained in the image data.

[0145] The target detection model is used to extract the position of the training image data to obtain the actual position information. The target loss function is determined according to the training position information and the actual position information. In this embodiment, the target loss function is preferably a cross entropy loss function. When the target loss function converges to a preset convergence threshold, it is determined that the second preset training end condition is met, and a language model is obtained.

[0146] In some embodiments, the target detection model is used to extract the position of the training image data. After obtaining the actual position information, the actual position information can also be normalized to convert data of different orders of magnitude into the same order of magnitude to eliminate the impact of the order of magnitude.

[0147] Based on the execution process of step 101 to step 104, in some embodiments, step 102 specifically includes:

[0148] Step 1021: input the image data into the language model, and obtain layout information corresponding to the image data through processing by the multimodal model in the language model.

[0149] Step 1022: perform recognition processing on the image data to obtain a plurality of initial target data corresponding to the image data.

[0150] Step 1023: Compare the initial target data with the text prompt information to obtain at least one target data.

[0151] Step 1024: extract from the layout information to obtain location information of at least one target data.

[0152] In a specific implementation, the image data is input into the language model, and processed by the multimodal model in the language model to obtain layout information corresponding to the image data. The image data is identified and processed to obtain a plurality of initial target data corresponding to the image data.

[0153] The number of initial target data obtained by identifying the image data may be more than the number of target data contained in the text prompt information. Therefore, the initial target data is compared with the text prompt information, and the target data that exists in the initial target data but does not exist in the text prompt information is deleted. The remaining initial target data is the target data.

[0154] Exemplarily, the initial target data includes a cat, a table, a chair and a dog, and the target data included in the text prompt information is a cat and a table, then the cat and the table are the target data corresponding to the image data.

[0155] The layout information is the position information of the entire image, and the position information of at least one target data is obtained by extracting from the layout information.

[0156] Through the above scheme, by comparing the initial target data with the text prompt information, unnecessary initial target data is eliminated, the amount of data is reduced, and the overall operating speed is improved.

[0157] In some embodiments, after obtaining the position information of at least one target data, the position information of each target data is converted so that the position information of each target data is adjusted to the same resolution size to facilitate subsequent feature extraction and stitching processing.

[0158] Based on the same inventive concept, another embodiment of the present disclosure provides an image generation method, which applies the image generation model described in the above embodiment, such as Figure 2 As shown, specifically including:

[0159] Step 201, obtaining text prompt information to be processed, and inputting the text prompt information to be processed into an image generation model.

[0160] Step 202: Identify the target data to be processed contained in the text prompt information to be processed, and use the image generation model to determine the target position information corresponding to the target data to be processed.

[0161] Step 203: Generate and output a target image corresponding to the text prompt information to be processed according to the target position information by using the image generation model.

[0162] During specific implementation, text prompt information to be processed is obtained, wherein the text prompt information to be processed is text prompt information input by a user, and the text prompt information to be processed is input into an image generation model, wherein the image generation model is a pre-trained model.

[0163] Identify the target data to be processed contained in the text prompt information to be processed, and use the image generation model to determine the target position information corresponding to the target data to be processed. Use the image generation model to generate and output a target image corresponding to the text prompt information to be processed according to the target position information, wherein the target image contains the target data to be processed, and the position of the target data to be processed is consistent with the target position information.

[0164] Through the above scheme, by using the image generation model to generate a target image consistent with the text prompt information to be processed, the determination of the target image is more accurate. At the same time, the target image contains the target data to be processed at the same position as that mentioned in the text prompt information to be processed, which can better meet the needs of users.

[0165] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0166] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0167] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a model training device.

[0168] refer to Figure 3 , Figure 3 A training device for the model of an embodiment includes:

[0169] The data acquisition module 301 is configured to acquire a training data set and an initial image generation model, wherein the training data set includes image data and text prompt information corresponding to the image data;

[0170] A location determination module 302 is configured to determine feature information of at least one target data in the image data;

[0171] A feature extraction module 303 is configured to extract features from the text prompt information and feature information of the at least one target data to obtain embedded features corresponding to the image data;

[0172] The training module 304 is configured to train the initial image generation model using the image data and the embedded features until a first preset training end condition is met to obtain an image generation model.

[0173] In some embodiments, the initial image generation model includes a text feature extraction model and an embedding model, and the feature extraction module 303 specifically includes:

[0174] A text embedding feature determination unit is configured to input the text prompt information into a text feature extraction model, and process the text prompt information through the text feature extraction model to obtain a text embedding feature corresponding to the image data;

[0175] a target embedding feature determination unit, configured to input feature information of the at least one target data into an embedding model, and obtain a target embedding feature corresponding to the image data through processing by the embedding model;

[0176] The embedding feature determination unit is configured to concatenate the text embedding feature and the target embedding feature to obtain the embedding feature corresponding to the image data.

[0177] In some embodiments, the text feature extraction model includes multiple initial text feature extraction models, and the text embedding feature determination unit is specifically configured as follows:

[0178] Inputting the text prompt information into a plurality of initial text feature extraction models respectively, and processing the text prompt information through the plurality of text feature extraction models to obtain a plurality of initial text embedding features;

[0179] All initial text embedding features are concatenated to obtain text embedding features corresponding to the image data.

[0180] In some embodiments, the embedding model includes a first embedding model and a second embedding model, and the target embedding feature determining unit is configured to:

[0181] Classify all target data to obtain category information corresponding to each target data in all target data;

[0182] Determine location information of at least one target data in the image data;

[0183] Inputting the category information corresponding to the target data into a first embedding model, and processing the first embedding model to obtain the category embedding features corresponding to the target data;

[0184] Inputting the position information corresponding to the target data into a second embedding model, and processing the second embedding model to obtain a coordinate embedding feature corresponding to the target data;

[0185] The category embedding feature and the coordinate embedding feature are added together to obtain an initial target embedding feature corresponding to the target data;

[0186] The initial target embedding features corresponding to all the target data are counted to obtain the target embedding features corresponding to the image data.

[0187] In some embodiments, the embedding feature determination unit is configured to:

[0188] Projecting the text embedding feature to a preset dimension to obtain the text embedding feature to be processed;

[0189] Projecting the target embedding feature to a preset dimension to obtain the target embedding feature to be processed;

[0190] Performing word segmentation processing on the text prompt information to obtain the number of word segments;

[0191] The target dimension is determined according to the number of word segments, and the embedding features of the text to be processed and the embedding features of the target to be processed are concatenated in the target dimension to obtain the embedding features corresponding to the image data.

[0192] In some embodiments, the apparatus further includes a training data set determination module, and the training data set determination module specifically includes:

[0193] A language model determination unit is configured to obtain training picture data, input the training picture data into an initial language model for training, and obtain a language model until a second preset training end condition is met;

[0194] a text prompt information determining unit, configured to input the image data into the language model, and obtain text prompt information corresponding to the image data through processing by the language model;

[0195] The data set determination unit is configured to aggregate the image data and the text prompt information corresponding to the image data to obtain a training data set.

[0196] In some embodiments, the initial language model includes a multimodal model and a target detection model, and the language model determination unit is specifically configured as follows:

[0197] Inputting the training image data into a multimodal model, and outputting training position information via the multimodal model;

[0198] Using the target detection model to extract the position of the training image data to obtain actual position information;

[0199] Determine a target loss function according to the training position information and the actual position information;

[0200] In response to the target loss function converging to a preset convergence threshold, it is determined that a second preset training end condition is met, and a language model is obtained.

[0201] In some embodiments, the location determination module 302 is specifically configured to:

[0202] Inputting the image data into the language model, and processing the image data through the multimodal model in the language model to obtain layout information corresponding to the image data;

[0203] Performing recognition processing on the image data to obtain a plurality of initial target data corresponding to the image data;

[0204] Compare the initial target data with the text prompt information to obtain at least one target data;

[0205] The location information of at least one target data is extracted from the layout information to obtain the location information.

[0206] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides an image generating device.

[0207] refer to Figure 4 , Figure 4 The image generating device of the embodiment comprises:

[0208] The data acquisition module 401 is configured to acquire the text prompt information to be processed and input the text prompt information to be processed into the image generation model;

[0209] The identification module 402 is configured to identify the target data to be processed contained in the text prompt information to be processed, and determine the target position information corresponding to the target data to be processed by using the image generation model;

[0210] The image generation module 403 is configured to generate and output a target image corresponding to the text prompt information to be processed according to the target position information by using the image generation model.

[0211] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0212] The device of the above embodiment is used to implement the training method of the corresponding model in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0213] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of the model described in any of the above embodiments is implemented.

[0214] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0215] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0216] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0217] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0218] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0219] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0220] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0221] The electronic device of the above-mentioned embodiment is used to implement the training method of the corresponding model in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0222] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the training method of the model described in any of the above embodiments.

[0223] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0224] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the model training method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0225] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a vehicle, including a training device for the model in the above-mentioned embodiments, an electronic device in the above-mentioned embodiments, and a computer-readable storage medium in the above-mentioned embodiments, wherein the vehicle equipment implements the training method for the model described in any of the above embodiments.

[0226] The vehicle of the above-mentioned embodiment is used to implement the training method of the model described in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0227] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0228] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0229] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0230] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0231] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0232] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0233] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0234] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the protection scope of the present disclosure.< / eos> < / eos>

Claims

1. A model training method, characterized in that: include: Acquire a training data set and an initial image generation model, wherein the training data set includes image data and text prompt information corresponding to the image data; Determine feature information of at least one target data in the image data; Extracting features from the text prompt information and the feature information of the at least one target data to obtain embedded features corresponding to the image data; The initial image generation model is trained using the picture data and the embedded features until a first preset training end condition is met, thereby obtaining an image generation model.

2. The method according to claim 1, characterized in that The initial image generation model includes a text feature extraction model and an embedding model. The extracting features of the text prompt information and the feature information of the at least one target data to obtain the embedded features corresponding to the image data includes: Inputting the text prompt information into a text feature extraction model, and processing it through the text feature extraction model to obtain text embedding features corresponding to the image data; Inputting feature information of the at least one target data into an embedding model, and processing the embedding model to obtain target embedding features corresponding to the image data; The text embedding feature and the target embedding feature are concatenated to obtain the embedding feature corresponding to the image data.

3. The method according to claim 2, characterized in that The embedding model includes a first embedding model and a second embedding model, The step of inputting the feature information of the at least one target data into the embedding model and processing the embedding model to obtain the target embedding feature corresponding to the image data includes: Classify all target data to obtain category information corresponding to each target data in all target data; Determine location information of at least one target data in the image data; Inputting the category information corresponding to the target data into a first embedding model, and processing the first embedding model to obtain the category embedding features corresponding to the target data; Inputting the position information corresponding to the target data into a second embedding model, and processing the second embedding model to obtain a coordinate embedding feature corresponding to the target data; The category embedding feature and the coordinate embedding feature are added together to obtain an initial target embedding feature corresponding to the target data; The initial target embedding features corresponding to all the target data are counted to obtain the target embedding features corresponding to the image data.

4. The method according to claim 2, characterized in that: The concatenating the text embedding feature and the target embedding feature to obtain the embedding feature corresponding to the image data includes: Projecting the text embedding feature to a preset dimension to obtain the text embedding feature to be processed; Projecting the target embedding feature to a preset dimension to obtain the target embedding feature to be processed; Performing word segmentation processing on the text prompt information to obtain the number of word segments; The target dimension is determined according to the number of word segments, and the embedding features of the text to be processed and the embedding features of the target to be processed are concatenated in the target dimension to obtain the embedding features corresponding to the image data.

5. The method according to claim 1, characterized in that: The process of determining the training data set includes: Acquire training picture data, input the training picture data into an initial language model for training, until a second preset training end condition is met, and obtain a language model; Inputting the image data into the language model, and processing the image data through the language model to obtain text prompt information corresponding to the image data; The image data and the text prompt information corresponding to the image data are aggregated to obtain a training data set.

6. The method according to claim 5, characterized in that The initial language model includes a multimodal model and a target detection model. The step of inputting the training picture data into the initial language model for training until a second preset training end condition is met to obtain a language model comprises: Inputting the training image data into a multimodal model, and outputting training position information via the multimodal model; Using the target detection model to extract the position of the training image data to obtain actual position information; Determine a target loss function according to the training position information and the actual position information; In response to the target loss function converging to a preset convergence threshold, it is determined that a second preset training end condition is met, and a language model is obtained.

7. The method according to claim 5, characterized in that The determining the location information of at least one target data in the image data includes: Inputting the image data into the language model, and processing the image data through the multimodal model in the language model to obtain layout information corresponding to the image data; Performing recognition processing on the image data to obtain a plurality of initial target data corresponding to the image data; Compare the initial target data with the text prompt information to obtain at least one target data; The location information of at least one target data is extracted from the layout information to obtain the location information.

8. An image generation method, characterized in that: Applying the image generation model described in any one of claims 1 to 7, comprising: Obtaining text prompt information to be processed, and inputting the text prompt information to be processed into an image generation model; Identifying the target data to be processed contained in the text prompt information to be processed, and determining the target position information corresponding to the target data to be processed by using the image generation model; The image generation model is used to generate and output a target image corresponding to the text prompt information to be processed according to the target position information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the program.

10. A vehicle, characterized in that: An electronic device comprising the electronic device described in claim 9.

Images