Visual data generation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of data processing, and provides a visual data generation method and device, electronic equipment and a readable storage medium. The method comprises the following steps: performing intention recognition on a visual data generation instruction to obtain structured data corresponding to the visual data generation instruction; task disassembling processing is conducted on the structured data, a visual data generation strategy is obtained, and the visual data generation strategy comprises a visual generation subtask and model information of a data generation model needed for executing the visual generation subtask; and according to the visual data generation strategy, the target visual data is generated based on the structured data, so that multi-modal data co-processing is realized by dynamically combining multiple generation models, the visual data generation efficiency and accuracy are improved, the multi-modal data co-processing capability in a complex scene is enhanced, and the multi-modal data processing efficiency is improved. And the semantic matching degree of the generated content and the user demand is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a visual data generation method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] The visual data generation technology generally adopts a paradigm of a single model directly responding to a generation instruction. Such a method generally processes the user instruction as a whole, does not sufficiently analyze the multi-dimensional elements in the complex intention, and thus causes semantic deviation or style mismatch between the generation result and the user demand.

[0003] Therefore, in the related art, the intention recognition is disengaged from the generation strategy planning, the cross-modal feature processing is insufficient, and the dynamic optimization mechanism is missing, which causes the problem of insufficient matching degree between the visual data generation result and the user demand. SUMMARY

[0004] Therefore, the embodiments of the present disclosure provide a visual data generation method and device, electronic equipment and readable storage medium to solve the problem of insufficient matching degree between the visual data generation result and the user demand due to the disengagement of the intention recognition from the generation strategy planning, the insufficient cross-modal feature processing and the missing dynamic optimization mechanism in the prior art.

[0005] In a first aspect, the embodiments of the present disclosure provide a visual data generation method, including: performing intention recognition on a visual data generation instruction to obtain structured data corresponding to the visual data generation instruction; performing task decomposition processing on the structured data to obtain a visual data generation strategy, wherein the visual data generation strategy includes a visual generation subtask and model information of a data generation model required for performing the visual generation subtask; and generating target visual data based on the structured data according to the visual data generation strategy.

[0006] In some embodiments, the visual data generation instruction includes to-be-processed instruction text data; and the intention recognition on the visual data generation instruction to obtain the structured data corresponding to the visual data generation instruction includes: performing intention analysis processing on the to-be-processed instruction text data to obtain element feature data corresponding to the to-be-processed instruction text data; and performing structured processing on the element feature data to obtain the structured data.

[0007] In some embodiments, the visual data generation instruction includes to-be-processed instruction text data and to-be-processed reference image data; the intent recognition is performed on the visual data generation instruction to obtain structured data corresponding to the visual data generation instruction, including: performing intent analysis processing on the to-be-processed instruction text data to obtain element feature data corresponding to the to-be-processed instruction text data; performing feature extraction processing on the to-be-processed reference image data to obtain image feature data corresponding to the to-be-processed reference image data; performing cross-modal text generation processing on the image feature data to obtain prompt text data corresponding to the to-be-processed reference image data; and performing structured processing on the element feature data, the image feature data, and the prompt text data to obtain the structured data.

[0008] In some embodiments, according to the visual data generation strategy, the target visual data is generated based on the structured data, including: calling a data generation model based on model information of the data generation model; performing parameter configuration on the data generation model based on a visual generation subtask to obtain an adaptive model corresponding to the visual data generation strategy; performing framework standardization processing on the structured data to obtain framework standardized data corresponding to the visual data generation instruction; and performing collaborative generation processing on the framework standardized data by the adaptive model to obtain the target visual data.

[0009] In some embodiments, the visual data generation strategy further includes a preset visual data similarity threshold; after the target visual data is generated based on the structured data according to the visual data generation strategy, the method further includes: performing fine-grained matching processing on the target visual data and the visual data generation instruction to obtain a similarity value corresponding to the target visual data; performing numerical comparison processing on the preset visual data similarity threshold and the similarity value corresponding to the target visual data; if the similarity value corresponding to the target visual data is greater than or equal to the preset visual data similarity threshold, the target terminal device is sent the target visual data for displaying the target visual data on a current graphical user interface of the target terminal device; and if the similarity value corresponding to the target visual data is less than the preset visual data similarity threshold, the visual data generation strategy is updated according to the similarity value corresponding to the target visual data to obtain a visual data generation update strategy; and the target visual data is updated based on the structured data according to the visual data generation update strategy.

[0010] In some embodiments, the framework standardization processing on the structured data to obtain the framework standardized data corresponding to the visual data generation instruction includes: performing format coding processing on the structured data by a preset standardization template to obtain a feature tensor corresponding to the visual data generation instruction; performing dimension alignment processing on the feature tensor to obtain an aligned tensor corresponding to the visual data generation instruction; and performing feature space normalization processing on the aligned tensor to obtain the framework standardized data.

[0011] In some embodiments, the framework standardized data is collaboratively generated by an adaptive model to obtain target visual data, including: performing feature coding processing on the framework standardized data to obtain a spatial feature map corresponding to the framework standardized data and a channel feature vector corresponding to the framework standardized data; performing geometric transformation alignment processing on the spatial feature map to obtain a target spatial feature map; and performing network fusion processing on the channel feature vector and the target spatial feature map by the adaptive model to obtain the target visual data.

[0012] In a second aspect of the embodiments of the present disclosure, a visual data generation apparatus is provided, including: a first processing module configured to perform intention recognition on a visual data generation instruction to obtain structured data corresponding to the visual data generation instruction; a second processing module configured to perform task decomposition processing on the structured data to obtain a visual data generation strategy, wherein the visual data generation strategy includes a visual generation subtask, model information of a data generation model required for executing the visual generation subtask, and a visual generation parameter; and a third processing module configured to generate target visual data based on the structured data according to the visual data generation strategy.

[0013] In a third aspect of the embodiments of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0014] In a fourth aspect of the embodiments of the present disclosure, a readable storage medium is provided, and the readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0015] Compared with the prior art, the embodiments of the present disclosure have the beneficial effects that: by responding to a visual data generation instruction and performing intention recognition on the visual data generation instruction to obtain structured data corresponding to the visual data generation instruction, the structured data is task-decomposed according to the information contained in the structured data to obtain a visual data generation strategy, and then the structured data can be processed according to the visual data generation strategy to obtain target visual data, so that multi-modal data collaborative processing is realized by dynamically combining multiple generation models, the generation efficiency and accuracy of visual data are improved, the collaborative processing capability of multi-modal data in complex scenarios is enhanced, and the semantic matching degree of generated content and user demand is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative effort.

[0017] Figure 1 is a scenario schematic diagram of an application scenario of an embodiment of the present disclosure;

[0018] Figure 2 is a flow schematic diagram of a visual data generation method provided by an embodiment of the present disclosure;

[0019] Figure 3 is a flow schematic diagram of another visual data generation method provided by an embodiment of the present disclosure;

[0020] Figure 4 is a structural schematic diagram of a visual data generation apparatus provided by an embodiment of the present disclosure;

[0021] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In the following description, specific details are set forth in order to provide a thorough understanding of embodiments of the present disclosure. However, persons skilled in the art will understand that embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known systems, structures, circuits, and methods have not been described in detail in order to avoid obscuring the description of the present disclosure.

[0023] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties. The visual data generation method can be implemented through a visual data generation model.

[0024] A visual data generation method and apparatus according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 is a scenario schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario can include terminal devices 1, 2, and 3, a server 4, and a network 5.

[0026] The terminal devices 1, 2 and 3 can be hardware or software. When the terminal devices 1, 2 and 3 are hardware, they can be various electronic devices with a display screen and supporting communication with the server 4, including but not limited to a smart phone, a tablet computer, a laptop computer and a desktop computer, etc.; when the terminal devices 1, 2 and 3 are software, they can be installed in the electronic devices as above. The terminal devices 1, 2 and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present disclosure do not make any limitation in this regard. Further, various applications can be installed on the terminal devices 1, 2 and 3, such as a data processing application, an instant messaging tool, a social platform software, a search application, a shopping application, etc.

[0027] The server 4 can be a server providing various services, for example, a background server receiving a request sent by a terminal device establishing a communication connection therewith. The background server can receive and analyze the request sent by the terminal device, etc., and generate a processing result. The server 4 can be a single server, a server cluster composed of several servers, or a cloud computing service center, and the embodiments of the present disclosure do not make any limitation in this regard.

[0028] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices providing various services for the terminal devices 1, 2 and 3. When the server 4 is software, it can be multiple software or software modules providing various services for the terminal devices 1, 2 and 3, or a single software or software module providing various services for the terminal devices 1, 2 and 3, and the embodiments of the present disclosure do not make any limitation in this regard.

[0029] The network 5 can be a wired network connected by coaxial cables, twisted pairs and optical fibers, or a wireless network realizing interconnection of various communication devices without wiring, for example, Bluetooth, Near Field Communication (NFC), Infrared, etc., and the embodiments of the present disclosure do not make any limitation in this regard.

[0030] A user can establish a communication connection with the server 4 via the network 5 through the terminal devices 1, 2 and 3 to receive or send information, etc. Specifically, the server 4 can obtain a visual data generation instruction via the terminal devices 1, 2 and 3, perform intent recognition on the visual data generation instruction, structure the visual data generation instruction to obtain structured data corresponding to the visual data generation instruction, perform task decomposition on the structured data according to information contained in the structured data to obtain a visual data generation strategy, and then perform processing on the structured data according to the visual data generation strategy to obtain target visual data.

[0031] It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure do not limit them.

[0032] Figure 2 is a flowchart of a visual data generation method provided by an embodiment of the present disclosure. Figure 2 The visual data generation method of the present disclosure can be executed by Figure 1 the server. As shown in Figure 2 , the visual data generation method includes:

[0033] In step S201, the intention of the visual data generation instruction is recognized to obtain the structured data corresponding to the visual data generation instruction.

[0034] Specifically, the visual data generation instruction can be parsed by a large language model through morphological analysis, syntactic analysis and semantic understanding, etc. without limitation, and then the core elements such as generation type, core theme, style requirement and special demand can be extracted without limitation. The obtained core elements can be structured according to the preset data format, and the structured data corresponding to the visual data generation instruction can be obtained, thereby enhancing the understanding ability of the visual data generation requirement, improving the accuracy and efficiency of the conversion from the visual data generation instruction to the structured data.

[0035] The visual data generation instruction can be a natural language description or command input by the user, and the visual data generation instruction can be used to instruct to generate specific visual data, for example, "generate a landscape picture containing blue sky, white clouds, grassland and a red balloon".

[0036] The structured data corresponding to the visual data generation instruction can be data obtained by organizing the information in the visual data generation instruction according to the predefined format and structure through the intention recognition processing of the visual data generation instruction. The structured data can be used for subsequent visual data generation process.

[0037] For example, in image generation, visual data generation instructions can be input into the visual data generation model, the visual data generation instructions are parsed by the large language model, and each word in the visual data generation instructions is recognized and obtained, such as "generate", "promotional poster", "background", "night", "city skyline", etc. Then the visual data generation instructions can be syntactically analyzed to determine the grammatical relationship between the words, for example, "the background is the city skyline at night" can be used to represent that "the city skyline at night" is the specific content of "background". Then semantic understanding can be performed to extract core elements, such as the background element can be "the city skyline at night" and there should be "flickering lights", the subject element can be "the latest model of the brand's mobile phone" and should be highlighted, and the text element can be "technology leading the future" and should be placed next to the center of the picture. Finally, the above elements are organized according to the structure of the lightweight data exchange format (JavaScript Object Notation, JSON) to generate structured data, for example, the structured data can include background element category, background element description, subject element category, subject element highlighting method, text element content, text element position, etc.

[0038] In step S202, the structured data is processed by task decomposition to obtain a visual data generation strategy, wherein the visual data generation strategy includes visual generation sub-tasks and model information of data generation models required for executing the visual generation sub-tasks.

[0039] Specifically, the structured data can be processed by a large language model with an inference-action framework, the relationships and dependencies between the elements in the structured data can be analyzed according to the logic and process of visual data generation, the structured data can be decomposed into multiple visual generation sub-tasks that are independent of each other but related to each other, including but not limited to generating an image according to visual data generation instructions, generating a series of images according to visual data generation instructions, or generating a picture-text report with images, text descriptions, and layout according to visual data generation instructions. Then, according to the characteristics and requirements of each visual generation sub-task, the data generation model required for executing the sub-task can be matched and determined from the pre-set model library, and the visual data generation strategy containing the visual generation sub-task and the model information of the corresponding data generation model can be obtained, thereby enhancing the decomposition and planning ability of complex visual data generation tasks, improving the orderliness and efficiency of task processing, and improving the quality and accuracy of visual data generation.

[0040] The visual data generation strategy can be a visual data generation scheme obtained by task decomposition processing of the structured data, and the visual data generation strategy can be used to represent each sub-task to be completed in the visual data generation process and the data generation model applied to execute each sub-task.

[0041] The visual generation subtask can be a task unit obtained by decomposing the visual data generation task, and each visual generation subtask can correspond to a link or target in the visual data generation process, for example, generating a background, drawing a main object, adding text annotations, etc., which is not limited herein.

[0042] The data generation model can be a learning model or algorithm for performing a visual generation subtask, which can generate corresponding visual content according to input parameters and data, including but not limited to an image generation model or a text rendering model, etc.

[0043] The model information of the data generation model can be detailed information for characterizing the data generation model, including but not limited to the type of the data generation model, the parameter setting of the data generation model, the input and output format of the data generation model, etc., which can be used to call and configure the data generation model in the visual data generation process.

[0044] For example, in the above application scenario, the generation of a promotional poster through the input visual data generation instruction can be decomposed into multiple visual generation subtasks, such as generating a city skyline background image at night, adding a flashing light effect on the background image, drawing an image of the latest mobile phone and ensuring its highlight, and rendering the text "Technology leads the future" next to the center of the picture. Then, according to the characteristics and requirements of each visual generation subtask, the data generation model and its model information required to perform the subtask can be matched and determined from the model library:

[0045] For example, a text-to-image model (Stable Diffusion) for basic generation, a neural network control model (ControlNet) for structure control, a text-to-image generation model (DeepFloyd) for detail enhancement, and / or an image super-resolution model (Enhanced Super-Resolution Generative Adversarial Network, Real-ESRGAN) for resolution enhancement, etc.

[0046] The above visual generation subtasks and corresponding model information jointly constitute the visual data generation strategy corresponding to the visual data generation instruction.

[0047] In step S203, the target visual data is generated based on the structured data according to the visual data generation strategy.

[0048] Specifically, the data generation model corresponding to each visual generation subtask can be called in turn according to the sequence of the visual generation subtasks. When the data generation model is called, the information in the structured data can be transmitted to the data generation model according to the input format preset by the model information, and then the data generation model can perform calculation and processing to generate the visual data segment corresponding to the visual generation subtask. The visual data segments generated by each visual generation subtask can be integrated and fused to obtain complete target visual data, thereby enhancing the flexibility and controllability of visual data generation, improving the automation degree and efficiency of the generation process, and improving the fit and quality of the target visual data to the user demand.

[0049] The target visual data can be visual content obtained by executing the visual data generation strategy, and can be used to represent visual content meeting the requirements of the user visual data generation instruction. The target visual data can be obtained by calling each data generation model to process the structured data and integrating and fusing the generated visual data segments. The form of the target visual data includes but is not limited to images, texts, and / or animations, etc. In addition, it should be noted that the target visual data can also include log data in the generation process.

[0050] For example, different generation strategies can be used for different task types according to the visual data generation strategy. For an image generation task based on a text description, a Stable Diffusion model can be used for basic generation. The input text description can be encoded and embedded by a text encoder to obtain a text feature vector, and a random Gaussian noise can be generated as an initial latent image representation. Then, in the latent space, the initial latent image representation can be gradually denoised, and the denoised latent image representation can be converted by a decoder to obtain a target image. For a series image generation task based on a text description, ControlNet can be used to perform semantic segmentation on the text description and generate corresponding series images. For example, in a task of generating a series of images of a panda eating bamboo, the key frames of the series images can be "morning mist bamboo forest", "midday meal", and "evening rest", respectively. A visual generation model can be used to generate images corresponding to the three key frames to obtain a target series image. For a text and image report generation task based on a text description, a data generation model can be used to generate images, and a visual large language model can be used to analyze the images to generate corresponding text descriptions. Then, the target image and the target text can be laid out and displayed.

[0051] According to the technical scheme provided by the embodiment of the present disclosure, the visual data generation instruction is parsed by the large language model to extract core elements and generate structured data. Then, the large language model with the reasoning-action framework can be used to decompose the task, dynamically select a data generation model, and obtain a step-by-step visual data generation strategy. Then, the combined data generation model can be called according to the visual data generation strategy to generate text-to-image, image-to-image, or multi-step collaborative generation, and obtain the target visual data. In this way, the semantic understanding depth and style transfer accuracy of multi-modal input are enhanced, the hierarchical planning ability and multi-model collaboration efficiency of complex generation tasks are improved, and the generation effect of the generation result in terms of theme consistency, style adaptability, and content integrity is improved.

[0052] In some embodiments, the visual data generation instruction includes to-be-processed instruction text data; the visual data generation instruction is subjected to intent recognition to obtain structured data corresponding to the visual data generation instruction, including: the to-be-processed instruction text data is subjected to intent analysis processing to obtain element feature data corresponding to the to-be-processed instruction text data; and the element feature data is subjected to structured processing to obtain the structured data.

[0053] Specifically, the to-be-processed instruction text data can be subjected to intent analysis processing by the large language model, which can be processed in the manner of task recognition, lexical analysis, syntactic analysis, and / or semantic understanding, without limitation, to analyze the semantic information of the to-be-processed instruction text data and extract elements related to visual data generation, including but not limited to task type, subject object, scene background, style requirement, and the like, to obtain element feature data corresponding to the to-be-processed instruction text data; and the element feature data can be organized and arranged according to a preset template to obtain the structured data.

[0054] The to-be-processed instruction text data can be text content input in natural language form, which can be used to instruct the generation of specific visual data, such as “generate a wedding scene photo in a retro style, including the couple, a floral arch, and friends and family, with a warm color tone”.

[0055] The element feature data corresponding to the to-be-processed instruction text data can be core element information extracted by intent analysis processing of the to-be-processed instruction text data, such as “the generation task is text-to-image”, the subject is “the couple, the floral arch, and friends and family”, the scene is “the wedding scene”, the style is “the retro style”, and the picture color tone is “warm”, without limitation. The element feature data can be used for subsequent structured processing.

[0056] For example, task recognition can be performed on the to-be-processed instruction text data, the to-be-processed instruction text data requires generating an image according to the text content, and then lexical analysis can be performed on the to-be-processed instruction text data, and the to-be-processed instruction text data is split into multiple words, such as “making”, “one”, “full of technological sense”, “city night scene picture” and the like; through syntactic analysis, the grammatical relationship between the words can be determined, for example, “the center of the picture is a high-rise glass curtain wall building” can be used to represent that “the center of the picture” is a locative adverbial, and “a high-rise glass curtain wall building” is the subject; and then semantic understanding can be performed to extract element feature data: the subject is “glass curtain wall building”, the scene is “city night scene”, the style is “full of technological sense”, the special effect is “there are flashing neon lights around the building” and “there is a meteor in the sky”, and the color tone is “blue and purple”.

[0057] The element feature data can be organized and arranged according to a preset structured template. For example, the subject, scene, style, special effect, color tone and the like are classified respectively, and the hierarchical relationship between them is clear, forming structured data.

[0058] According to the technical scheme provided by the embodiments of the present disclosure, the to-be-processed instruction text data is processed by the large language model in the manner of task recognition, lexical analysis, syntactic analysis and / or semantic understanding, the semantic information of the to-be-processed instruction text data is analyzed, and the elements related to visual data generation are extracted, and then the element feature data corresponding to the to-be-processed instruction text data is obtained; the element feature data can be organized and arranged according to a preset template to obtain structured data, thereby enhancing the understanding accuracy of the visual data generation requirement, improving the conversion efficiency from natural language instruction to structured data, and improving the pertinence and accuracy of the subsequent visual data generation process.

[0059] In some embodiments, the visual data generation instruction includes to-be-processed instruction text data and to-be-processed reference image data; the visual data generation instruction is subjected to intent recognition to obtain structured data corresponding to the visual data generation instruction, including: performing intent analysis processing on the to-be-processed instruction text data to obtain element feature data corresponding to the to-be-processed instruction text data; performing feature extraction processing on the to-be-processed reference image data to obtain image feature data corresponding to the to-be-processed reference image data; performing cross-modal text generation processing on the image feature data to obtain prompt text data corresponding to the to-be-processed reference image data; and performing structured processing on the element feature data, the image feature data and the prompt text data to obtain the structured data.

[0060] Specifically, the big language model can be used to analyze the intent of the instruction text data to be processed, and the corresponding element feature data can be obtained. The multi-modal neural network model (Contrastive Language-Image Pre-Training, CLIP) can be used to extract features from the reference image data to be processed, including color, texture, shape, etc. The image feature data can be obtained by the cross-modal text generation model (Bootstrapping Language-Image Pre-training, BLIP), which can convert the image feature data into natural language representation prompt text data. The element feature data, image feature data, and prompt text data can be organized and arranged according to the preset structured template to obtain structured data.

[0061] The reference image data to be processed can be an image sample used to assist in generating target visual data. The reference image data to be processed can provide a reference for the style, composition, and / or color of the target visual data to be generated, without limitation. For example, when generating an artistic painting, a work of a painter can be input as a reference image to represent that the generated painting can have similar style and expression.

[0062] The image feature data corresponding to the reference image data to be processed can be the internal feature information of the reference image data to be processed extracted by the CLIP model, such as color distribution, texture feature, shape feature, and spatial layout, without limitation. The image feature data can be used to analyze the style and characteristics of the reference image data to be processed.

[0063] The prompt text data corresponding to the reference image data to be processed can be the natural language description text obtained by converting the image feature data corresponding to the reference image data to be processed through the BLIP model. The prompt text data can summarize the features and style of the reference image data to be processed in the form of text, such as "this image has a warm color tone, the center of the picture is a circular object, surrounded by irregular lines, and the overall style is abstract art." The prompt text data can be used to assist in understanding the features of the reference image data to be processed.

[0064] For example, the instruction text data to be processed can be "design a clothing poster with a retro style, the clothing is mainly a dress, the color is deep red and gold, and the picture should have an elegant atmosphere", and the reference image data to be processed can be a fashion magazine cover, on which the model wears retro-style clothing, and the picture has soft color tone and simple composition.

[0065] The text "design a clothing poster with a retro style, mainly a dress, with deep red and gold color matching, and an elegant atmosphere" can be analyzed by a large language model for lexical analysis, syntactic analysis and semantic understanding, and the core elements can be extracted to obtain element feature data: the task is to generate a single image, the style is "retro style", the main body is "dress", the color is "deep red and gold color matching", and the atmosphere is "elegant".

[0066] The CLIP model can be used to analyze the fashion magazine cover image to extract image feature data, such as color features, texture features, shape features, and spatial layout features.

[0067] The BLIP model can be used to generate prompt text data based on image features, such as "this image presents a retro fashion style, the picture is dominated by warm colors, the model wears a dress with lace and satin elements, the posture is elegant, and the picture layout is simple and elegant".

[0068] The element feature data, image feature data, and prompt text data can be structured, and the prompt text can be used as auxiliary understanding information to obtain structured data.

[0069] According to the technical scheme provided by the embodiments of the present disclosure, the large language model is used to analyze the intent of the instruction text data to be processed, and the corresponding element feature data is obtained. The CLIP model can be used to extract features from the reference image data to be processed to obtain image feature data. The BLIP model can be used to convert the image feature data into natural language representation prompt text data. The element feature data, image feature data, and prompt text data can be organized and arranged according to the preset structured template to obtain structured data. This enhances the understanding of the multi-modal visual data generation instruction, improves the conversion accuracy and comprehensiveness from text and image information to structured data, and improves the quality of subsequent visual data generation and the matching degree with user needs.

[0070] In some embodiments, according to the visual data generation strategy, the target visual data is generated based on the structured data, including: calling a data generation model based on model information of the data generation model; configuring parameters of the data generation model based on a visual generation subtask to obtain an adaptive model corresponding to the visual data generation strategy; performing frame standardization processing on the structured data to obtain frame standardized data corresponding to the visual data generation instruction; and performing collaborative generation processing on the frame standardized data through the adaptive model to obtain the target visual data.

[0071] Specifically, the model information of the data generation model included in the visual data generation strategy can be used to call a corresponding data generation model from a preset model library. The data generation model obtained by calling can be configured according to the requirements of the visual generation subtask, for example, the resolution, style parameters, and / or color mode of the generated image can be adjusted, and the like, without limitation, to obtain an adaptive model corresponding to the visual data generation strategy. The structured data can be standardized according to the input data framework and format requirements of each data generation model, and each element in the structured data can be sorted and converted to obtain framework standardized data corresponding to the visual data generation instruction. The framework standardized data can be input into the adaptive model for collaborative generation processing.

[0072] The adaptive model corresponding to the visual data generation strategy can be a model obtained by configuring the parameters of the called data generation model according to the visual generation subtask. The adaptive model can adapt to the requirements of the current visual generation subtask, for example, when generating an illustration of a specific style, the adaptive model can adjust the style parameters and line thickness, etc., to make the generated illustration more accurate in style and performance. The adaptive model can be obtained by optimizing and configuring the data generation model according to the requirements of the visual generation subtask. The adaptive model can be used to generate visual data segments or complete target visual data that meet the requirements of a specific subtask.

[0073] The framework standardized data corresponding to the visual data generation instruction can be data obtained by standardizing the framework of the structured data. The framework standardized data can sort and convert each element in the structured data according to the input data framework and format requirements of the adaptive model, for example, different formats of color information can be unified into a specific encoding format, shape descriptions can be converted into standard geometric parameter representations, and the like, without limitation, to ensure that the data meets the input specifications of the adaptive model. The framework standardized data can be used as input data for the adaptive model.

[0074] For example, the model information of the data generation model included in the visual data generation strategy can be used to call a corresponding data generation model from a preset model library, such as a Stable Diffusion model and a ControlNet model. The resolution, style parameters, and color mode of the generated image can be adjusted to obtain an adaptive model corresponding to the visual data generation strategy. The structured data can be standardized according to the input data framework and format requirements of each data generation model, and each element in the structured data can be sorted and converted to obtain framework standardized data corresponding to the visual data generation instruction. The framework standardized data can be input into the adaptive model for collaborative generation processing to obtain the target visual data.

[0075] According to the technical scheme provided by the embodiment of the present disclosure, the model information of the data generation model contained in the visual data generation strategy is generated according to the visual data, the corresponding data generation model is called from the preset model library, the data generation model obtained by calling is configured according to the requirements of the visual generation subtask, the adaptive model corresponding to the visual data generation strategy is obtained, the structured data can be standardized according to the framework, the elements in the structured data are sorted and converted according to the input data framework and format requirements of each data generation model, the framework standardized data corresponding to the visual data generation instruction is obtained, the framework standardized data is input into the adaptive model, the data is processed by the adaptive model, and the target visual data is obtained. In this way, the adaptability and flexibility of the model in the visual data generation process are enhanced, the processing capability and generation effect of the visual data generation model for different visual generation subtasks are improved, and the quality of the target visual data and the matching degree with the user demand are improved.

[0076] In some embodiments, the visual data generation strategy further includes a preset visual data similarity threshold; after generating the target visual data based on the structured data according to the visual data generation strategy, the method further includes: performing fine-grained matching processing on the target visual data and the visual data generation instruction to obtain a similarity value corresponding to the target visual data; performing numerical comparison processing on the preset visual data similarity threshold and the similarity value corresponding to the target visual data; if the similarity value corresponding to the target visual data is greater than or equal to the preset visual data similarity threshold, the target visual data is sent to the target terminal device, and the target visual data is displayed on the current graphical user interface of the target terminal device; if the similarity value corresponding to the target visual data is less than the preset visual data similarity threshold, the visual data generation strategy is updated according to the similarity value corresponding to the target visual data to obtain a visual data generation update strategy; and the target visual data is updated based on the structured data according to the visual data generation update strategy.

[0077] Specifically, the multi-dimensional data such as image content and semantic information of the target visual data can be matched with the demand information contained in the visual data generation instruction through fine-grained matching to obtain a similarity value corresponding to the target visual data. The similarity value can be compared with a preset visual data similarity threshold value. If the similarity value corresponding to the target visual data is greater than or equal to the preset visual data similarity threshold value, the generated target visual data meets the demand, and the target visual data can be sent to the target terminal device and displayed on the current graphical user interface of the target terminal device. If the similarity value is less than the preset threshold value, the model parameters or generation rules in the visual data generation strategy can be adjusted and optimized according to the gap between the similarity value and the preset visual data similarity threshold value to obtain a visual data generation update strategy. Then, the target visual data can be regenerated based on the original structured data according to the updated visual data generation strategy.

[0078] The preset visual data similarity threshold value can be a standard value preset in the visual data generation strategy. The preset visual data similarity threshold value can be used to represent the similarity between the generated target visual data and the demand expressed by the visual data generation instruction. The preset visual data similarity threshold value can be set to 0.5, 0.8, or 0.95, etc. When the similarity between the generated target visual data and the instruction demand is greater than or equal to the preset visual data similarity threshold value, the generated image meets the requirements.

[0079] The similarity value corresponding to the target visual data can be a value calculated by multi-dimensional analysis and calculation of the generated target visual data and the visual data generation instruction through fine-grained matching processing. The similarity value can be used to represent the similarity between the target visual data and the instruction demand. Fine-grained matching processing can be comparing each feature of the target visual data with the corresponding demand feature in the visual data generation instruction, calculating the similarity through cosine similarity or Euclidean distance algorithm, etc. Then, the similarity value corresponding to the target visual data can be obtained by integrating the similarity of each dimension. The similarity value corresponding to the target visual data can be used to compare with the preset visual data similarity threshold value to determine whether the target visual data meets the requirements.

[0080] The target terminal device can be a device for receiving and displaying the generated target visual data, including but not limited to a smartphone, a tablet computer, a computer display, or a smart television, etc. The target terminal device can include at least one graphical user interface.

[0081] The current graphical user interface can be an interface running on the target terminal device for interaction with the user. The current graphical user interface can contain various graphical elements, including but not limited to display areas such as buttons, menus, icons, and / or text boxes, and can be used to display target visual data.

[0082] The visual data generation update strategy can be a new generation strategy obtained by adjusting and optimizing the original visual data generation strategy according to the difference between the similarity value and the preset visual data similarity threshold when the similarity value corresponding to the target visual data is less than the preset visual data similarity threshold. The visual data generation update strategy can include adjusting the parameters of the data generation model, changing the data preprocessing method, or updating the generation rules and other information, which are not limited here.

[0083] For example, in an e-commerce platform, a product display image needs to be generated for a pair of women's shoes, and the preset visual data similarity threshold can be 0.75. The image content and semantic information of the target visual data are processed in a fine-grained manner according to the visual data generation instructions, such as analyzing whether the color of the women's shoes in the generated image is black, whether the style is high-heeled sandals, whether the position and style of the gold decoration meet the requirements, and whether the background is simple white, etc. By calculating the similarity of the color histogram, the matching degree of the object contour, and the consistency of the semantic label, etc., the similarity value corresponding to the target visual data is 0.7.

[0084] Comparing the similarity value 0.7 corresponding to the target visual data with the preset visual data similarity threshold 0.75, since 0.7 is less than 0.75, there is a gap between the generated target visual data and the requirements of the visual data generation instructions, and the visual data generation strategy can be updated according to the difference between the similarity value and the preset visual data similarity threshold. For example, although the color of the generated women's shoes is black, the color tone is dark, which does not match the bright black required in the visual data generation instructions. Therefore, the color parameters of the image generation model can be adjusted in the visual data generation update strategy to make the color of the generated women's shoes brighter.

[0085] According to the visual data generation update strategy, the target visual data is regenerated based on the original structured data, and then fine-grained matching processing is performed again. The updated similarity value is 0.8, which is greater than the preset visual data similarity threshold. Therefore, the new target visual data can be sent to the server of the e-commerce platform and displayed on the product page.

[0086] According to the technical scheme provided by the embodiment of the present disclosure, by fine-grained matching, the multi-dimensional data such as image content and semantic information of the target visual data is matched with the requirement information contained in the visual data generation instruction, and a similarity value corresponding to the target visual data is obtained; the similarity value can be compared with a preset visual data similarity threshold value; if the similarity value corresponding to the target visual data is greater than or equal to the preset visual data similarity threshold value, the generated target visual data meets the requirement, and the target visual data can be sent to the target terminal device and displayed on the current graphical user interface of the target terminal device; if the similarity value is less than the preset threshold value, the model parameters or generation rules in the visual data generation strategy can be adjusted and optimized according to the gap between the similarity value and the preset visual data similarity threshold value, and a visual data generation update strategy is obtained; then, the target visual data can be regenerated based on the original structured data according to the updated visual data generation strategy, thereby enhancing the quality control capability of the visual data generation process, improving the degree of coincidence between the generated target visual data and the user requirement, and improving the user experience and the reliability of the visual data generation system.

[0087] In some embodiments, the structured data is subjected to framework standardization processing to obtain framework standardized data corresponding to the visual data generation instruction, including: the structured data is subjected to format coding processing through a preset standardization template to obtain a feature tensor corresponding to the visual data generation instruction; the feature tensor is subjected to dimension alignment processing to obtain an aligned tensor corresponding to the visual data generation instruction; and the aligned tensor is subjected to feature space normalization processing to obtain the framework standardized data.

[0088] Specifically, the structured data can be subjected to framework standardization processing, the structured data can be converted into a feature tensor corresponding to the visual data generation instruction through format coding processing of the structured data by a preset standardization template; then, the feature tensor can be subjected to dimension alignment processing to obtain an aligned tensor corresponding to the visual data generation instruction; and the aligned tensor can be subjected to feature space normalization to obtain the framework standardized data.

[0089] The feature tensor corresponding to the visual data generation instruction can be a data form obtained by format coding processing of the structured data by the preset standardization template. The feature tensor can be a tensor form in which the structured data is coded into a multi-dimensional structure, and the feature tensor can be used to represent the feature information of the structured data.

[0090] The aligned tensor corresponding to the visual data generation instruction can be a data form obtained by dimension alignment processing of the feature tensor. The aligned tensor can be obtained by adjusting the dimensions of the feature tensor so that the feature tensor has a unified dimension structure between different data samples.

[0091] For example, in the field of medical image analysis, medical images differ in format, resolution, and / or dimension, and the like. The structured data corresponding to the medical image data can be format-coded by a preset standardization template, the feature information corresponding to the medical image data is encoded to obtain a corresponding feature tensor. For example, for a computed tomography (CT) image, the feature tensor can include pixel values, image sizes, and / or slice thicknesses, and the like. The feature tensor can then be subjected to dimension alignment processing to unify the dimension structure of the feature tensor and obtain a corresponding aligned tensor. The aligned tensor can be subjected to feature space normalization processing to scale the medical image data to a preset range and obtain framework standardized data.

[0092] According to the technical scheme provided by the embodiments of the present disclosure, the structured data is subjected to framework standardization processing. The structured data is format-coded by a preset standardization template to convert the structured data into a feature tensor corresponding to the visual data generation instruction. The feature tensor is subjected to dimension alignment processing to obtain an aligned tensor corresponding to the visual data generation instruction. The aligned tensor is subjected to feature space normalization to obtain framework standardized data. In this way, the compatibility and comparability between visual data from different sources are enhanced, and the efficiency and accuracy of data processing and analysis are improved.

[0093] In some embodiments, the framework standardized data is subjected to collaborative generation processing by an adaptive model to obtain target visual data, including: the framework standardized data is subjected to feature encoding processing to obtain a spatial feature map corresponding to the framework standardized data and a channel feature vector corresponding to the framework standardized data; the spatial feature map is subjected to geometric transformation alignment processing to obtain a target spatial feature map; the channel feature vector and the target spatial feature map are subjected to network fusion processing by the adaptive model to obtain the target visual data.

[0094] Specifically, the framework standardized data can be subjected to feature encoding processing to extract a spatial feature map corresponding to the framework standardized data and a channel feature vector corresponding to the framework standardized data by specific encoding. The spatial feature map can be subjected to geometric transformation alignment processing by translation, rotation, scaling, and the like, without limitation, to achieve a target spatial layout and obtain a target spatial feature map. The channel feature vector and the target spatial feature map can be fused by splicing or weighted fusion of the adaptive model, without limitation, to obtain the target visual data.

[0095] The spatial feature map corresponding to the framework standardized data can be a two-dimensional or multi-dimensional matrix obtained by feature encoding processing on the framework standardized data. The spatial feature map can be used to represent the feature distribution of the framework standardized data in the spatial dimension. The feature encoding processing can be extracted from the framework standardized data through a feature encoding structure, including but not limited to convolution layers and pooling layers in a convolutional neural network.

[0096] The channel feature vector corresponding to the framework standardized data can be a vector obtained by feature encoding processing on the framework standardized data. The channel feature vector can be used to represent the feature intensity or importance of the framework standardized data in different channels. The channel feature vector can be extracted from the framework standardized data through a feature encoding structure. Each element in the channel feature vector represents a feature value of a channel, and the element can be used to represent the contribution degree of the channel to the overall data.

[0097] The target spatial feature map can be a result obtained by geometric transformation alignment processing on the spatial feature map corresponding to the framework standardized data. The geometric transformation can be performed by translation, rotation, and / or scaling of the spatial feature map, which eliminates the differences in space between different data sources.

[0098] For example, in the field of autonomous driving, visual data collected by a camera or radar sensor installed on a vehicle is obtained through framework standardization processing to obtain framework standardized data. The image data collected by the camera can be encoded by a convolutional neural network to obtain a spatial feature map. The spatial feature map can be used to represent the distribution and position relationship of objects in the image in space. Then, a channel feature vector can be extracted through a fully connected layer. The channel feature vector can be used to represent the contribution information of different color channels or feature channels to the image.

[0099] The spatial feature map can be processed by geometric transformation alignment. The spatial feature map can be rotated and translated to align the spatial feature map with the coordinate system of the vehicle to obtain a target spatial feature map. Then, the channel feature vector and the target spatial feature map can be processed by network fusion through an adaptive model. The channel feature vector and the target spatial feature map can be weighted and fused to obtain target visual data. The target visual data can be used to represent the environmental information around the vehicle.

[0100] According to the technical scheme provided by the embodiment of the present disclosure, the frame standardized data is processed by feature coding, and the spatial feature map and the channel feature vector corresponding to the frame standardized data are extracted through specific coding respectively. Then, the spatial feature map can be processed by geometric transformation alignment to make the spatial feature map reach the target spatial layout and obtain the target spatial feature map. The channel feature vector and the target spatial feature map can be fused through the adaptive model to obtain the target visual data, thereby enhancing the fusion degree and consistency of different data sources at the feature level, improving the representation ability of the target visual data to the original information, and improving the accuracy and reliability of the target visual data.

[0101] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0102] Figure 3 is a flowchart of another visual data generation method provided by the embodiment of the present disclosure. As shown in Figure 3 the visual data generation method comprises:

[0103] User input, including image (to-be-processed reference image data) and text description (to-be-processed instruction text data);

[0104] Instruction intention recognition, including that the visual data generation instruction can be recognized for intention, and the to-be-processed instruction text data can be parsed for text, and then the to-be-processed reference image data can be extracted for image features through the CLIP model, the core elements extracted can be stored in the context information through the generation of a counter-question sentence, and the structured information can be generated in response to the confirmation instruction;

[0105] Strategy planning, including task decomposition based on the ReAct intelligent agent (reasoning-action large model), model selection based on the subtasks (visual generation subtasks) obtained by decomposition, obtaining model information of a data generation model required for executing the visual generation subtask, and pre-setting a CLIP model similarity threshold (a preset visual data similarity threshold);

[0106] Image generation engine, including taking the strategy (visual data generation strategy) and the context information obtained by the above planning as input data, generating a style prompt word (frame standardized data) based on a visual large model according to reference image features, generating an image through multi-model cooperation, calling different models (data generation models) for generation according to the strategy, and then saving the images generated at each step;

[0107] Image evaluation, including fine-grained evaluation processes such as semantic matching, style consistency evaluation, image quality evaluation, and content integrity evaluation, etc. dimensions, and then according to the evaluation dimensions, the defect positioning can be carried out, and the positioned defect problem can be input into the large language model to output the generated problem and subsequent suggestions;

[0108] Quality determination and strategy optimization, including inputting evaluation results and strategy planning results as input data, comparing image evaluation scores with preset visual data similarity thresholds, if the score does not meet the standard, re-generating the image, automatically adjusting the generation parameters based on the defect type, if the score meets the standard, integrating the results, outputting the target visual data and all intermediate step generation logs.

[0109] For example, another process provided by the visual data generation method can include:

[0110] 1. Instruction intention recognition

[0111] Input: "Generate 3 ink-style panda eating bamboo pictures" and URL or local file

[0112] Output: "JSON format structured task description"

[0113] (1) Text analysis: analyze user intent through a large language model (such as GPT-4) and extract core elements:

[0114] Generation type (single picture / series picture / picture-text combination)

[0115] Core theme (such as "panda eating bamboo")

[0116] Style requirements (such as ink / cyberpunk)

[0117] Special needs (such as "highlight dynamic")

[0118] (2) Image feature extraction (if there is a reference picture):

[0119] Extract image feature vector (512-dimensional numerical value) using CLIP model

[0120] Generate a textual description of the image (such as "ink-style bamboo forest") using the BLIP model

[0121] (3) Intention verification:

[0122] Generate a counter-question sentence to confirm details (such as "Do you want to keep the reference picture's blur effect?")

[0123] After the user confirms, store the features and styles in the context

[0124] Finally, get structured information (structured generation)

[0125] Output

[0126]

[0127] 2. Strategy Planning

[0128] Input: Structured task description + historical generation data

[0129] Output: Hierarchical generation strategy (including steps, model selection, evaluation criteria)

[0130] Processing logic:

[0131] (1) Task decomposition:

[0132] Use large language models in the ReAct framework for step-by-step planning:

[0133] Single-image generation according to whether there is a reference image. If not, choose text-to-image; if there is, choose image-to-image series. Break down the series of images into key frames (e.g. "morning mist bamboo forest → noon meal → dusk rest")

[0134] Break down the image-text report into "image generation + text description + layout design"

[0135] (2) Model selection:

[0136] Dynamic combination model:

[0137] Basic generation: Stable Diffusion

[0138] Structure control: ControlNet

[0139] Detail enhancement: DeepFloyd

[0140] Super-resolution: Real-ESRGAN

[0141] (3) Quality pre-evaluation:

[0142] Set CLIP score threshold (landscape painting ≥ 0.75)

[0143] Set the upper limit of the number of iterations (default 3 times)

[0144] Output

[0145]

[0146] 3. Image generation engine

[0147] Input: Generation strategy (steps, models, parameters), context information (reference image features, user confirmed details)

[0148] Output: Intermediate / final image list

[0149] Processing logic:

[0150] (1) Dynamic parameter adjustment:

[0151] Automatic generation of style prompt words based on visual large models according to reference image features

[0152] Without reference images, generate style prompt words only based on input text

[0153] Generate style prompt words based on reference image features, such as "impressionist style, warm palette"

[0154] (2) Multi-model collaboration:

[0155] Call different models according to strategy steps:

[0156] Text-to-image, image-to-image (single image generation): execute according to planning steps

[0157] Series of images: iteratively generated based on key frames

[0158] Image-text report: first generate images according to planned steps, then use visual large language model to supplement related text information based on images.

[0159] (3) Intermediate result caching:

[0160] Save the generated images at each step

[0161] Used for subsequent tracing and repeated generation

[0162] Output

[0163]

[0164] 4. Image evaluation

[0165] Input: generated image, original prompt, reference image features (optional)

[0166] Output: optimization instructions (JSON)

[0167] Processing logic:

[0168] (1) Fine-grained evaluation:

[0169] Semantic matching: CLIP score (≥0.8)

[0170] Style consistency: StyleCLIP score (≥0.85)

[0171] Image quality: FID score (≤15)

[0172] Content integrity: Visual Large Model (VLM) detects missing elements

[0173] (2) Defect localization

[0174] CLIP score less than threshold → theme deviation

[0175] Image generation model (StyleCLIP) score less than threshold → style inconsistency

[0176] Deep learning generation model evaluation indicator (FID) score less than threshold → image quality defect

[0177] Insufficient content integrity

[0178] (3) Feedback suggestion generation

[0179] Input the localized defect problem into the large language model, output the generated problem and subsequent suggestions, such as: Problem: missing background elements; suggestion: add "shimmering lake surface" to enhance dynamic feeling

[0180] Output

[0181]

[0182] 5. Quality determination and optimization

[0183] Input: evaluation results + strategy planning results

[0184] (1) Determination logic:

[0185] Score meets standards → enter result integration

[0186] Does not meet standards → automatically retry, go to image generation module, re-perform image generation (up to 3 times)

[0187] (2) Automatically adjust generation parameters based on defect type:

[0188] If "theme deviation" → increase "exact representation of", switch to MidJourney model.

[0189] If "style inconsistency" → force add "in the style of [reference figure]"; increase style weight to 1.2.

[0190] If "image blur" → enable Real-ESRGAN super resolution; increase generation steps to 75.

[0191] If "content missing" -> add specific element cue (e.g. "lush bamboo forest"); enable ControlNet completion.

[0192] (3) Context backtracking:

[0193] If multiple failures, return to strategy planning module to re-plan image generation strategy.

[0194] Output

[0195]

[0196] 6. Result integration

[0197] Input: final generated image + generation log (including all intermediate steps)

[0198] Output: multi-format deliverables

[0199] Processing logic:

[0200] (1) Multi-format output:

[0201] Single image: directly return high-resolution image

[0202] Series of images: generate HTML5 animation with timeline, or directly output multiple images

[0203] Image-text report: based on large models, automatically generate Markdown image-text documents

[0204] (2) Generation traceability:

[0205] Generate JSON metadata file (including all intermediate step records)

[0206] (3) User feedback

[0207] Embed quick feedback button (e.g. "style mismatch")

[0208] Automatically record user preferences for model fine-tuning

[0209] Output

[0210]

[0211] The following is an embodiment of the device of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the device embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0212] Figure 4 is a schematic diagram of a visual data generation device provided by an embodiment of the present disclosure. As shown in Figure 4 , the visual data generation device comprises:

[0213] The first processing module 401 is configured to perform intent recognition on the visual data generation instruction to obtain structured data corresponding to the visual data generation instruction.

[0214] The second processing module 402 is configured to perform task decomposition processing on the structured data to obtain a visual data generation strategy, wherein the visual data generation strategy includes a visual generation subtask and model information of a data generation model required for performing the visual generation subtask.

[0215] The third processing module 403 is configured to generate target visual data based on the structured data according to the visual data generation strategy.

[0216] According to the technical scheme provided by the embodiments of the present disclosure, the visual data generation instruction is responded to, the visual data generation instruction is subjected to intent recognition, and the structured data corresponding to the visual data generation instruction is obtained by structuring the visual data generation instruction. The structured data is subjected to task decomposition according to the information contained in the structured data to obtain a visual data generation strategy. Then, the structured data can be processed according to the visual data generation strategy to obtain target visual data. In this way, the multi-modal data collaborative processing is realized by dynamically combining multiple generation models. The data compatibility is improved by using the feature tensor dimension alignment and space normalization technology. The generation efficiency and accuracy of the visual data are improved. The collaborative processing capability of the multi-modal data in a complex scene is enhanced. The semantic matching degree of the generated content and the user demand is improved.

[0217] In some embodiments, the first processing module 401 is specifically configured to perform intent analysis processing on the to-be-processed instruction text data to obtain element feature data corresponding to the to-be-processed instruction text data; and perform structured processing on the element feature data to obtain the structured data.

[0218] In some embodiments, the first processing module 401 is specifically configured to perform intent analysis processing on the to-be-processed instruction text data to obtain element feature data corresponding to the to-be-processed instruction text data; perform feature extraction processing on the to-be-processed reference image data to obtain image feature data corresponding to the to-be-processed reference image data; perform cross-modal text generation processing on the image feature data to obtain prompt text data corresponding to the to-be-processed reference image data; and perform structured processing on the element feature data, the image feature data, and the prompt text data to obtain the structured data.

[0219] In some embodiments, the third processing module 403 is specifically configured to invoke the data generation model based on the model information of the data generation model; perform parameter configuration on the data generation model based on the visual generation subtask to obtain an adaptive model corresponding to the visual data generation strategy; perform framework standardization processing on the structured data to obtain framework standardized data corresponding to the visual data generation instruction; and perform collaborative generation processing on the framework standardized data through the adaptive model to obtain the target visual data.

[0220] In some embodiments, the above-mentioned visual data generating device is also used to perform fine-grained matching processing on the target visual data and the visual data generation instruction to obtain a similarity value corresponding to the target visual data; perform numerical comparison processing on the preset visual data similarity threshold and the similarity value corresponding to the target visual data; if the similarity value corresponding to the target visual data is greater than or equal to the preset visual data similarity threshold, then send the target visual data to the target terminal device for displaying the target visual data on the current graphical user interface of the target terminal device; if the similarity value corresponding to the target visual data is less than the preset visual data similarity threshold, then update the visual data generation strategy according to the similarity value corresponding to the target visual data to obtain a visual data generation update strategy; according to the visual data generation update strategy, update the target visual data based on the structured data.

[0221] In some embodiments, the structured data is subjected to framework normalization processing to obtain framework normalized data corresponding to the visual data generation instruction, which is specifically used to perform format encoding processing on the structured data through a preset standardized template to obtain a feature tensor corresponding to the visual data generation instruction; perform dimension alignment processing on the feature tensor to obtain an alignment tensor corresponding to the visual data generation instruction; and perform feature space normalization processing on the alignment tensor to obtain framework normalized data.

[0222] In some embodiments, the framework normalized data is collaboratively generated and processed through an adaptation model to obtain target visual data, which is specifically used to perform feature encoding processing on the framework normalized data to obtain a spatial feature map corresponding to the framework normalized data and a channel feature vector corresponding to the framework normalized data; perform geometric transformation and alignment processing on the spatial feature map to obtain a target spatial feature map; and perform network fusion processing on the channel feature vector and the target spatial feature map through an adaptation model to obtain target visual data.

[0223] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0224] Figure 5 Schematic diagram of the electronic device 5 provided by the embodiment of the present disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable by the processor 501. When the processor 501 executes the computer program 503, the steps of the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0225] The electronic device 5 can be an electronic device such as a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The electronic device 5 can include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art can understand that the processor 501 and the memory 502 can be divided into different functional units or modules according to the actual needs, and the functions of the processor 501 and the memory 502 can be combined into one processing unit, or two or more units can be integrated into one unit. Figure 5 The electronic device 5 is only an example and does not constitute a limitation on the electronic device 5, and can include more or fewer components or different components than those shown.

[0226] The processor 501 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like.

[0227] The memory 502 can be an internal storage unit of the electronic device 5, such as a hard disk or a memory of the electronic device 5. The memory 502 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. The memory 502 can also include both the internal storage unit and the external storage device of the electronic device 5. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0228] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit or module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit, and the integrated unit can be realized in the form of hardware or software.

[0229] The integrated modules / units, if implemented in the form of software functional units and sold or used as independent products, can be stored in a readable storage medium (for example, a computer readable storage medium). Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program can include computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable storage medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0230] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure.

Claims

1. A method for generating visual data, characterized in that: include: Performing intent recognition on the visual data generation instruction to obtain structured data corresponding to the visual data generation instruction; Performing task decomposition processing on the structured data to obtain a visual data generation strategy, wherein the visual data generation strategy includes visual generation subtasks and model information of a data generation model required to execute the visual generation subtasks; According to the visual data generation strategy, target visual data is generated based on the structured data.

2. The method for generating visual data according to claim 1, wherein: The visual data generation instruction includes instruction text data to be processed; The performing intent recognition on the visual data generation instruction to obtain structured data corresponding to the visual data generation instruction includes: Performing intent analysis on the instruction text data to be processed to obtain element feature data corresponding to the instruction text data to be processed; The element feature data is structured to obtain the structured data.

3. The method for generating visual data according to claim 1, wherein: The visual data generation instruction includes instruction text data to be processed and reference image data to be processed; The performing intent recognition on the visual data generation instruction to obtain structured data corresponding to the visual data generation instruction includes: Performing intent analysis on the instruction text data to be processed to obtain element feature data corresponding to the instruction text data to be processed; Performing feature extraction processing on the reference image data to be processed to obtain image feature data corresponding to the reference image data to be processed; Performing cross-modal text generation processing on the image feature data to obtain prompt text data corresponding to the reference image data to be processed; The element feature data, the image feature data and the prompt text data are structured to obtain the structured data.

4. The method for generating visual data according to claim 1, wherein: Generating target visual data based on the structured data according to the visual data generation strategy includes: Based on the model information of the data generation model, calling the data generation model; Based on the visual generation subtask, parameter configuration is performed on the data generation model to obtain an adaptation model corresponding to the visual data generation strategy; Performing framework standardization processing on the structured data to obtain framework standardized data corresponding to the visual data generation instruction; The framework standardized data is collaboratively generated and processed by the adaptation model to obtain the target visual data.

5. The method for generating visual data according to claim 1, wherein: The visual data generation strategy also includes a preset visual data similarity threshold; After generating target visual data based on the structured data according to the visual data generation strategy, the method further includes: Performing fine-grained matching processing on the target visual data and the visual data generation instruction to obtain a similarity value corresponding to the target visual data; Performing numerical comparison processing on a preset visual data similarity threshold and a similarity value corresponding to the target visual data; If the similarity value corresponding to the target visual data is greater than or equal to the preset visual data similarity threshold, sending the target visual data to the target terminal device for displaying the target visual data on the current graphical user interface of the target terminal device; If the similarity value corresponding to the target visual data is less than the preset visual data similarity threshold, updating the visual data generation strategy according to the similarity value corresponding to the target visual data to obtain a visual data generation update strategy; An update strategy is generated according to the visual data, and the target visual data is updated based on the structured data.

6. The method for generating visual data according to claim 4, wherein: The performing framework normalization processing on the structured data to obtain framework normalized data corresponding to the visual data generation instruction includes: Performing format encoding processing on the structured data using a preset standardized template to obtain a feature tensor corresponding to the visual data generation instruction; Performing dimension alignment processing on the feature tensor to obtain an alignment tensor corresponding to the visual data generation instruction; Performing feature space normalization processing on the alignment tensor to obtain the frame normalized data.

7. The method for generating visual data according to claim 4, wherein: The collaborative generation processing of the framework standardized data by the adaptation model to obtain the target visual data includes: Performing feature encoding processing on the framework-normalized data to obtain a spatial feature map corresponding to the framework-normalized data and a channel feature vector corresponding to the framework-normalized data; Performing geometric transformation and alignment processing on the spatial feature map to obtain a target spatial feature map; The channel feature vector and the target space feature map are subjected to network fusion processing through the adaptation model to obtain the target visual data.

8. A visual data generating device, characterized in that: include: A first processing module is configured to perform intent recognition on a visual data generation instruction and obtain structured data corresponding to the visual data generation instruction; a second processing module, configured to perform task decomposition processing on the structured data to obtain a visual data generation strategy, wherein the visual data generation strategy includes visual generation subtasks and model information of a data generation model required to execute the visual generation subtasks; The third processing module is configured to generate target visual data based on the structured data according to the visual data generation strategy.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images