Data processing method, device, apparatus and readable storage medium

Abstract

The application discloses a data processing method, device and equipment and a readable storage medium. The method comprises the following steps: acquiring generated demand content input by an object; performing task division on the generated demand content by a task division agent component to obtain N sub-tasks with an execution sequence; acquiring an unexecuted sub-task S i from the N sub-tasks in the execution agent component according to the execution sequence i ; generating an execution instruction according to input parameters and model calling interface information corresponding to the sub-task S i ; calling a processing model corresponding to the model calling interface information in a model library according to the execution instruction, processing the input parameters by using the processing model, and obtaining an output result of the sub-task S i . When the sub-task S is the Nth sub-task, the output result of the Nth sub-task is determined as media data comprising a plurality of split shot contents indicated by the generated demand content. According to the application, media data meeting task requirements can be efficiently and low-costly generated.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and readable storage medium. Background Technology

[0002] After a community channel is created, it needs a cold start, which involves populating the channel with content. This can be done by manually publishing content or by purchasing content from other communities. However, the number of channels and the amount of content that can be manually published are very limited, and the labor costs are high with low production efficiency. Purchasing content is also difficult to apply to all channels due to its high cost. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and readable storage medium that can efficiently and cost-effectively generate media data that meets task requirements.

[0004] This application provides a data processing method, which is executed by a business server. The business server includes a task partitioning proxy component and an execution proxy component; including:

[0005] Obtain the generated requirements from the object input;

[0006] The generated requirements are divided into N subtasks with an execution order by a task partitioning proxy component. The N subtasks are then submitted to the execution proxy component; N is a positive integer.

[0007] In the execution proxy component, retrieve the unexecuted subtasks S from the N subtasks according to the execution order. i , where i is a positive integer less than or equal to N;

[0008] Using a large language model to analyze subtask S i Perform model invocation and recognition to obtain subtask S i Corresponding model call interface information;

[0009] According to subtask S i The corresponding input parameters and model API information are used to generate execution instructions. Based on the execution instructions, the corresponding processing model in the model library is called. The processing model processes the input parameters to obtain subtask S. i The output results; Subtask S i The corresponding input parameters are determined based on the output results or generated requirements of one or more executed subtasks;

[0010] When subtask S iWhen it is the Nth subtask, the output of the Nth subtask is determined as the media data including multiple storyboard content indicated by the generated content requirements; the output of the Nth subtask is obtained by integrating the material data from the output of the previous N-1 subtasks.

[0011] The business server also includes an analytics proxy component; and also includes:

[0012] The media data is submitted to the analytics proxy component by executing the proxy component;

[0013] The quality score is obtained by analyzing the media data through proxy components.

[0014] If the quality score is greater than or equal to the score threshold, the media data will be delivered to the platform.

[0015] If the quality score is less than the score threshold, data modification suggestions will be generated for the media data;

[0016] If the data modification feedback indicates that the task partitioning proxy component is abnormal in dividing the generated requirements into tasks, then the data modification feedback will be submitted to the task partitioning proxy component.

[0017] If the data modification comments indicate an anomaly in the model selection for the model library, the data modification comments will be submitted to the execution agent component.

[0018] The media data includes storyboard images and dialogue text for each storyboard scene; a quality score is obtained by analyzing the media data using a proxy component, including:

[0019] By analyzing the dialogue style of the dialogue text in each scene by the proxy component, the dialogue style type is obtained.

[0020] Feature extraction is performed on the storyboard images to obtain image features, and the image style type of the storyboard images is identified based on the image features;

[0021] An image quality score is generated based on the matching relationship between image features, dialogue style type, and image style type.

[0022] Semantic coherence is identified in the dialogue text of each scene to obtain a dialogue quality score;

[0023] A quality score for the media data is generated based on the dialogue quality score and the image quality score.

[0024] Specifically, the task partitioning proxy component divides the generated requirements into tasks, resulting in N subtasks with an execution order, including:

[0025] The task partitioning proxy component identifies the task intent of the generated requirement content to obtain N task intents.

[0026] Obtain the story generation intent from N task intents, construct story generation prompts for the story generation intents, and generate story generation subtasks containing the story generation prompts;

[0027] Obtain storyboard generation intents from N task intents, construct storyboard generation prompts for the storyboard generation intents, determine the first dependency relationship between the input parameters in the storyboard generation prompts and the story generation subtasks, and generate storyboard generation subtasks containing storyboard generation prompts and the first dependency relationship;

[0028] Obtain the image generation intent from N task intents, determine the second dependency between the input parameters required by the image generation intent and the storyboard generation subtask, and generate an image generation subtask that includes the image generation intent and the second dependency.

[0029] Obtain the integration intent from N task intents, determine the third dependency relationship between the input parameters required for the integration intent and the storyboard generation subtask and the image generation subtask, and generate an integration subtask containing the integration intent and the third dependency relationship.

[0030] Based on the first dependency, second dependency, and third dependency, the story generation subtask, storyboard generation subtask, image generation subtask, and integration subtask are identified as N subtasks with an execution order.

[0031] The execution agent component is configured with a task pipeline; within the execution agent component, unexecuted subtasks S are retrieved from N subtasks according to the execution order. i ,include:

[0032] When the task pipeline determines that none of the N subtasks have been executed, the task pipeline retrieves the subtask S with the highest execution priority among the N subtasks. i ,

[0033] When subtask S is obtained through the task pipeline i-1 When returning the output, the result is obtained through the task pipeline from subtask S. i-1 The execution order of adjacent and unexecuted subtasks S i ;

[0034] According to subtask S i The corresponding input parameters and model call interface information generate execution instructions, including:

[0035] When subtask S i When executing the highest-priority subtask, generate subtask S based on the generated requirements. iThe corresponding input parameters, based on subtask S i The corresponding input parameters and model call interface information are used to generate execution instructions;

[0036] When subtask S i If the subtask is not the highest priority subtask in the execution order, the subtask S is determined based on the output results of one or more already executed subtasks. i The corresponding input parameters, based on subtask S i Based on the corresponding input parameters and model call interface information, an execution instruction is generated; one or more executed subtasks include subtask S. i-1 .

[0037] The N subtasks include story generation, storyboard generation, image generation, and integration; based on subtask S... i The corresponding input parameters and model call interface information generate execution instructions, including:

[0038] If subtask S i For story generation subtasks, the story generation prompts and the first API call information in the story generation subtask are encapsulated into execution instructions; the first API call information is the model API call information corresponding to the story generation subtask.

[0039] If subtask S i For the storyboard generation subtask, the story text content from the output of the story generation subtask is input into the storyboard generation prompt words in the storyboard generation subtask to obtain the updated storyboard generation prompt words. The updated storyboard generation prompt words and the second call interface information are encapsulated into an execution instruction. The execution order of the storyboard generation subtask is after the execution order of the story generation subtask. The second call interface information is the model call interface information corresponding to the storyboard generation subtask.

[0040] If subtask S i For the image generation subtask, the storyboard description text and the third calling interface information in the output of the storyboard generation subtask are encapsulated into execution instructions; the execution order of the image generation subtask is after the execution order of the storyboard generation subtask; the third calling interface information is the model calling interface information corresponding to the image generation subtask.

[0041] If subtask S i To integrate subtasks, the input parameters of the integrated subtasks are obtained, and execution instructions are generated based on the input parameters of the integrated subtasks and the fourth calling interface information. The input parameters of the integrated subtasks are determined based on the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask. The execution order of the integrated subtasks follows the execution order of the image generation subtasks. The fourth calling interface information is the model calling interface information corresponding to the integrated subtasks.

[0042] Specifically, based on the execution instruction, the corresponding processing model is selected from the model library according to the model call interface information. The input parameters are then processed by the processing model to obtain subtask S. i The output includes:

[0043] If subtask S i For the sub-task of storyboard generation, the storyboard generation model corresponding to the second calling interface information is selected from the model library according to the execution instruction corresponding to the sub-task of storyboard generation;

[0044] The story text content in the updated storyboard generation prompts is broken down using a storyboard generation model to obtain the split text content corresponding to multiple storyboards.

[0045] Identify the main subject used for dialogue in each segmented text content, and generate a storyboard object description text for the main subject of each storyboard based on each segmented text content;

[0046] Generate storyboard dialogue text for each scene based on the content of each segmented text;

[0047] The story script content is analyzed for plot style using a storyboard generation model to obtain style analysis information.

[0048] Generate background description text for each scene based on the content of each segmented text;

[0049] The storyboard object description text, storyboard dialogue text, style analysis information, and storyboard background description text are determined as the output of the storyboard generation subtask.

[0050] The storyboard description text includes storyboard object description text, storyboard dialogue text, and storyboard background description text; based on the execution instruction, the corresponding processing model is selected from the model library according to the model call interface information, and the input parameters are processed by the processing model to obtain subtask S. i The output includes:

[0051] If subtask S i For image generation subtasks, the image generation model corresponding to the third calling interface information is selected from the model library according to the execution instructions corresponding to the image generation subtask.

[0052] The storyboard description text in the execution instruction is input into the image generation model. The image generation model fuses and encodes the storyboard object description text and the storyboard dialogue text to obtain the main text encoding vector. The storyboard dialogue text and the storyboard background description text are fused and encoded to obtain the background text encoding vector.

[0053] The noisy image is acquired and input into the forward diffusion network layer in the image generation model. The forward diffusion network layer outputs a forward noise vector.

[0054] The forward noise vector, the main text encoding vector, and the background text encoding vector are input into the back diffusion network layer in the image generation model. In the back diffusion network layer, the forward noise vector is denoised by the main text encoding vector to obtain the main image that matches the storyboard object description text and the storyboard dialogue text.

[0055] In the back-diffusion network layer, the forward noise vector is denoised by the background text encoding vector to obtain a background image that matches the storyboard dialogue text and the storyboard background description text.

[0056] The main image and background image are determined as the output of the image generation subtask.

[0057] The storyboard description text includes storyboard dialogue and style analysis information, while the source images include main images and background images. Based on the execution instructions, the corresponding processing model is selected from the model library using the model call interface information. The input parameters are then processed by the processing model to obtain subtask S. i The output includes:

[0058] If subtask S i To integrate subtasks, the integration model corresponding to the fourth call interface information is selected from the model library through the execution instructions corresponding to the integration subtasks, and the storyboard dialogue text, style analysis information, main image and background image are input into the integration model as input parameters;

[0059] The subject image is cropped and modified by integrating the model to obtain the corrected subject image. The background image is then redrawn and modified based on the image features of the corrected subject image to obtain the corrected background image.

[0060] Based on the style analysis information, the dialogue style of the storyboard dialogue text is identified, a dialogue text box with the dialogue style is created, and the storyboard dialogue text is filled into the dialogue text box to obtain the dialogue embedding information;

[0061] The main image and the embedded dialogue information are fused into the background image to obtain the output of the integration subtask.

[0062] Among them, the subtask S is performed using a large language model. i Model selection and recognition are performed to obtain subtask S. i The corresponding model call interface information includes:

[0063] In the execution agent component, through subtask S iThe task intent is to identify the first data type of the required input parameters and the second data type of the expected output result;

[0064] Subtask S i The corresponding first and second data types are input into the large language model, and the large language model is used to process subtask S. i Perform model invocation and recognition to obtain subtask S i Corresponding model call interface information; Subtask S i The corresponding model call interface information indicates that the corresponding processing model supports input data of the first data type and output data of the second data type.

[0065] This also includes:

[0066] Obtain the meta-task data template and action sequence data; the action sequence data includes sample input parameters and sample output results generated by the historical model based on the sample input parameters; the meta-task data template refers to the question template associated with the processing model in the model library;

[0067] Based on the data type of the sample input parameters, the model attributes of the historical model, and the data type of the sample output results, the meta-task data template is constructed into question sample data. The question sample data is input into the initial large language model, and the predicted answer data corresponding to the question sample data is output through the initial large language model.

[0068] Training loss values ​​are generated based on action sequence data and predicted response data. The model parameters of the initial large language model are then adjusted based on these training loss values ​​to obtain the parameters suitable for subtask S. i Large language models for model call recognition.

[0069] Among them, the N subtasks also include enhancement subtasks; the model call interface information corresponding to the enhancement subtasks is the fifth call interface information; and also includes:

[0070] If subtask S i To enhance the subtask, the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask are determined as the input parameters for the enhancement subtask. The integration subtask is executed after the enhancement subtask, and the enhancement subtask is executed after the image generation subtask.

[0071] The input parameters and fifth call interface information of the enhanced subtask are encapsulated into the execution instructions of the enhanced subtask;

[0072] Based on the execution instruction, the model corresponding to the model call interface information is selected from the model library. The input parameters are then processed by the processing model to obtain subtask S. iThe output includes:

[0073] If subtask S i To enhance the subtask, the enhanced model corresponding to the fifth calling interface information is obtained from the model library through the execution instructions of the enhanced subtask, and the input parameters of the enhanced subtask are input into the enhanced model.

[0074] The input parameters of the augmentation subtask are extracted by the augmentation model to obtain input feature data. Based on the input feature data, feature retrieval is performed in the knowledge base to obtain candidate feature augmentation data.

[0075] Similarity is calculated for the input parameters of the candidate feature enhancement data and the enhancement subtask to obtain the feature enhancement similarity. Candidate feature enhancement data with a feature enhancement similarity greater than the similarity threshold are identified as feature enhancement data.

[0076] The input parameters of the augmentation subtask are augmented based on the feature augmentation data to obtain the output result of the augmentation subtask; the augmented storyboard description text and augmented source images in the output result of the augmentation subtask are used as input parameters for the integration subtask.

[0077] Among them, the N subtasks also include evaluation subtasks; the model call interface information corresponding to the evaluation subtasks is the sixth call interface information; and also includes:

[0078] If subtask S i To evaluate the subtask, the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask are determined as the input parameters for the evaluation subtask. The execution order of the integration subtask is after the execution order of the evaluation subtask, and the execution order of the evaluation subtask is after the execution order of the image generation subtask.

[0079] The input parameters of the evaluation subtask and the information of the sixth calling interface are encapsulated into the execution instructions of the evaluation subtask;

[0080] Based on the execution instruction, the model corresponding to the model call interface information is selected from the model library. The input parameters are then processed by the processing model to obtain subtask S. i The output includes:

[0081] If subtask S i To evaluate the subtask, the evaluation model corresponding to the sixth call interface information is obtained from the model library through the execution instructions of the evaluation subtask, and the input parameters of the evaluation subtask are input into the evaluation model.

[0082] The storyboard description text is extracted using an evaluation model to obtain its features. The source image is also extracted using the same model to obtain its features. Finally, the storyboard description text features and the source image features are fused using a multimodal feature fusion method to obtain a multimodal feature vector.

[0083] The quality score is calculated on the multimodal feature vectors to obtain the quality score result. If the quality score result is greater than or equal to the score threshold, style recognition is performed on the multimodal feature vectors to obtain the style category.

[0084] If the style category matches the media data style category indicated by the generated content requirements, then the storyboard description text and source images are determined as the output of the evaluation subtask; the storyboard description text and source images in the output of the evaluation subtask are used as input parameters for the integration subtask.

[0085] One embodiment of this application provides a data processing apparatus, including:

[0086] The proxy component processing module is used to obtain the generation requirements of the object input;

[0087] The proxy component processing module is also used to divide the generated requirements into tasks through the task partitioning proxy component, resulting in N sub-tasks with an execution order, and submitting the N sub-tasks to the execution proxy component; N is a positive integer;

[0088] The proxy component processing module is also used to retrieve unexecuted subtasks S from N subtasks in the execution order within the proxy component. i , where i is a positive integer less than or equal to N;

[0089] The proxy component processing module is also used to process subtasks S through a large language model. i Perform model invocation and recognition to obtain subtask S i Corresponding model call interface information;

[0090] The proxy component processing module is also used to process subtasks S. i The corresponding input parameters and model call interface information are used to generate execution instructions;

[0091] The model processing module is used to call the corresponding processing model from the model library according to the execution instructions, and process the input parameters through the processing model to obtain the subtask S. i The output results; Subtask S i The corresponding input parameters are determined based on the output results or generated requirements of one or more executed subtasks;

[0092] The proxy component processing module is also used when subtask S iWhen it is the Nth subtask, the output of the Nth subtask is determined as the media data including multiple storyboard content indicated by the generated content requirements; the output of the Nth subtask is obtained by integrating the material data from the output of the previous N-1 subtasks.

[0093] In one possible implementation, the business server also includes an analysis proxy component; the proxy component processing module is further used to perform the following operations:

[0094] The media data is submitted to the analytics proxy component by executing the proxy component;

[0095] The quality score is obtained by analyzing the media data through proxy components.

[0096] If the quality score is greater than or equal to the score threshold, the media data will be delivered to the platform.

[0097] If the quality score is less than the score threshold, data modification suggestions will be generated for the media data;

[0098] If the data modification feedback indicates that the task partitioning proxy component is abnormal in dividing the generated requirements into tasks, then the data modification feedback will be submitted to the task partitioning proxy component.

[0099] If the data modification comments indicate an anomaly in the model selection for the model library, the data modification comments will be submitted to the execution agent component.

[0100] In one possible implementation, the media data includes storyboard images and dialogue text for each storyboard scene; the proxy component processing module is used to perform quality assessment of the media data by analyzing the proxy component and obtaining a quality score, specifically by performing the following operations:

[0101] By analyzing the dialogue style of the dialogue text in each scene by the proxy component, the dialogue style type is obtained.

[0102] Feature extraction is performed on the storyboard images to obtain image features, and the image style type of the storyboard images is identified based on the image features;

[0103] An image quality score is generated based on the matching relationship between image features, dialogue style type, and image style type.

[0104] Semantic coherence is identified in the dialogue text of each scene to obtain a dialogue quality score;

[0105] A quality score for the media data is generated based on the dialogue quality score and the image quality score.

[0106] In one possible implementation, the proxy component processing module is used to divide the generated requirements into N subtasks with an execution order by the task partitioning proxy component. Specifically, it performs the following operations:

[0107] The task partitioning proxy component identifies the task intent of the generated requirement content to obtain N task intents.

[0108] Obtain the story generation intent from N task intents, construct story generation prompts for the story generation intents, and generate story generation subtasks containing the story generation prompts;

[0109] Obtain storyboard generation intents from N task intents, construct storyboard generation prompts for the storyboard generation intents, determine the first dependency relationship between the input parameters in the storyboard generation prompts and the story generation subtasks, and generate storyboard generation subtasks containing storyboard generation prompts and the first dependency relationship;

[0110] Obtain the image generation intent from N task intents, determine the second dependency between the input parameters required by the image generation intent and the storyboard generation subtask, and generate an image generation subtask that includes the image generation intent and the second dependency.

[0111] Obtain the integration intent from N task intents, determine the third dependency relationship between the input parameters required for the integration intent and the storyboard generation subtask and the image generation subtask, and generate an integration subtask containing the integration intent and the third dependency relationship.

[0112] Based on the first dependency, second dependency, and third dependency, the story generation subtask, storyboard generation subtask, image generation subtask, and integration subtask are identified as N subtasks with an execution order.

[0113] In one possible implementation, the execution proxy component is configured with a task pipeline; the proxy component processing module is used to retrieve unexecuted subtasks S from N subtasks in the execution proxy component according to the execution order. i Specifically, it is used to perform the following operations:

[0114] When the task pipeline determines that none of the N subtasks have been executed, the task pipeline retrieves the subtask S with the highest execution priority among the N subtasks. i ,

[0115] When subtask S is obtained through the task pipeline i-1 When returning the output, the result is obtained through the task pipeline from subtask S. i-1 The execution order of adjacent and unexecuted subtasks S i ;

[0116] The proxy component processing module is used to process subtasks S i When generating execution instructions based on the corresponding input parameters and model call interface information, they are specifically used to perform the following operations:

[0117] When subtask S i When executing the highest-priority subtask, generate subtask S based on the generated requirements. i The corresponding input parameters, based on subtask S i The corresponding input parameters and model call interface information are used to generate execution instructions;

[0118] When subtask S i If the subtask is not the highest priority subtask in the execution order, the subtask S is determined based on the output results of one or more already executed subtasks. i The corresponding input parameters, based on subtask S i Based on the corresponding input parameters and model call interface information, an execution instruction is generated; one or more executed subtasks include subtask S. i-1 .

[0119] In one possible implementation, the N subtasks include a story generation subtask, a storyboard generation subtask, an image generation subtask, and an integration subtask; the proxy component processing module is used to process the subtasks S... i When generating execution instructions based on the corresponding input parameters and model call interface information, they are specifically used to perform the following operations:

[0120] If subtask S i For story generation subtasks, the story generation prompts and the first API call information in the story generation subtask are encapsulated into execution instructions; the first API call information is the model API call information corresponding to the story generation subtask.

[0121] If subtask S i For the storyboard generation subtask, the story text content from the output of the story generation subtask is input into the storyboard generation prompt words in the storyboard generation subtask to obtain the updated storyboard generation prompt words. The updated storyboard generation prompt words and the second call interface information are encapsulated into an execution instruction. The execution order of the storyboard generation subtask is after the execution order of the story generation subtask. The second call interface information is the model call interface information corresponding to the storyboard generation subtask.

[0122] If subtask S i For the image generation subtask, the storyboard description text and the third calling interface information in the output of the storyboard generation subtask are encapsulated into execution instructions; the execution order of the image generation subtask is after the execution order of the storyboard generation subtask; the third calling interface information is the model calling interface information corresponding to the image generation subtask.

[0123] If subtask S i To integrate subtasks, the input parameters of the integrated subtasks are obtained, and execution instructions are generated based on the input parameters of the integrated subtasks and the fourth calling interface information. The input parameters of the integrated subtasks are determined based on the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask. The execution order of the integrated subtasks follows the execution order of the image generation subtasks. The fourth calling interface information is the model calling interface information corresponding to the integrated subtasks.

[0124] In one possible implementation, the model processing module is used to select the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0125] If subtask S i For the sub-task of storyboard generation, the storyboard generation model corresponding to the second calling interface information is selected from the model library according to the execution instruction corresponding to the sub-task of storyboard generation;

[0126] The story text content in the updated storyboard generation prompts is broken down using a storyboard generation model to obtain the split text content corresponding to multiple storyboards.

[0127] Identify the main subject used for dialogue in each segmented text content, and generate a storyboard object description text for the main subject of each storyboard based on each segmented text content;

[0128] Generate storyboard dialogue text for each scene based on the content of each segmented text;

[0129] The story script content is analyzed for plot style using a storyboard generation model to obtain style analysis information.

[0130] Generate background description text for each scene based on the content of each segmented text;

[0131] The storyboard object description text, storyboard dialogue text, style analysis information, and storyboard background description text are determined as the output of the storyboard generation subtask.

[0132] In one possible implementation, the storyboard description text includes storyboard object description text, storyboard dialogue text, and storyboard background description text; the model processing module is used to select the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0133] If subtask Si For image generation subtasks, the image generation model corresponding to the third calling interface information is selected from the model library according to the execution instructions corresponding to the image generation subtask.

[0134] The storyboard description text in the execution instruction is input into the image generation model. The image generation model fuses and encodes the storyboard object description text and the storyboard dialogue text to obtain the main text encoding vector. The storyboard dialogue text and the storyboard background description text are fused and encoded to obtain the background text encoding vector.

[0135] The noisy image is acquired and input into the forward diffusion network layer in the image generation model. The forward diffusion network layer outputs a forward noise vector.

[0136] The forward noise vector, the main text encoding vector, and the background text encoding vector are input into the back diffusion network layer in the image generation model. In the back diffusion network layer, the forward noise vector is denoised by the main text encoding vector to obtain the main image that matches the storyboard object description text and the storyboard dialogue text.

[0137] In the back-diffusion network layer, the forward noise vector is denoised by the background text encoding vector to obtain a background image that matches the storyboard dialogue text and the storyboard background description text.

[0138] The main image and background image are determined as the output of the image generation subtask.

[0139] In one possible implementation, the storyboard description text includes storyboard dialogue text and style analysis information, and the source images include main images and background images; the model processing module is used to select the processing model corresponding to the model call interface information in the model library according to the execution instructions, and process the input parameters through the processing model to obtain subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0140] If subtask S i To integrate subtasks, the integration model corresponding to the fourth call interface information is selected from the model library through the execution instructions corresponding to the integration subtasks, and the storyboard dialogue text, style analysis information, main image and background image are input into the integration model as input parameters;

[0141] The subject image is cropped and modified by integrating the model to obtain the corrected subject image. The background image is then redrawn and modified based on the image features of the corrected subject image to obtain the corrected background image.

[0142] Based on the style analysis information, the dialogue style of the storyboard dialogue text is identified, a dialogue text box with the dialogue style is created, and the storyboard dialogue text is filled into the dialogue text box to obtain the dialogue embedding information;

[0143] The main image and the embedded dialogue information are fused into the background image to obtain the output of the integration subtask.

[0144] In one possible implementation, the proxy component processing module is used to process subtasks S through a large language model. i Model selection and recognition are performed to obtain subtask S. i When calling the corresponding model's API, it is specifically used to perform the following operations:

[0145] In the execution agent component, through subtask S i The task intent is to identify the first data type of the required input parameters and the second data type of the expected output result;

[0146] Subtask S i The corresponding first and second data types are input into the large language model, and the large language model is used to process subtask S. i Perform model invocation and recognition to obtain subtask S i Corresponding model call interface information; Subtask S i The corresponding model call interface information indicates that the corresponding processing model supports input data of the first data type and output data of the second data type.

[0147] In one possible implementation, the proxy component processing module is also used to perform the following operations:

[0148] Obtain the meta-task data template and action sequence data; the action sequence data includes sample input parameters and sample output results generated by the historical model based on the sample input parameters; the meta-task data template refers to the question template associated with the processing model in the model library;

[0149] Based on the data type of the sample input parameters, the model attributes of the historical model, and the data type of the sample output results, the meta-task data template is constructed into question sample data. The question sample data is input into the initial large language model, and the predicted answer data corresponding to the question sample data is output through the initial large language model.

[0150] Training loss values ​​are generated based on action sequence data and predicted response data. The model parameters of the initial large language model are then adjusted based on these training loss values ​​to obtain the parameters suitable for subtask S. i Large language models for model call recognition.

[0151] In one possible implementation, the N subtasks further include enhancement subtasks; the model call interface information corresponding to the enhancement subtasks is the fifth call interface information; the proxy component processing module is also used to perform the following operations:

[0152] If subtask S i To enhance the subtask, the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask are determined as the input parameters for the enhancement subtask. The integration subtask is executed after the enhancement subtask, and the enhancement subtask is executed after the image generation subtask.

[0153] The input parameters and fifth call interface information of the enhanced subtask are encapsulated into the execution instructions of the enhanced subtask;

[0154] The model processing module is used to select the corresponding processing model from the model library based on the model call interface information according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0155] If subtask S i To enhance the subtask, the enhanced model corresponding to the fifth calling interface information is obtained from the model library through the execution instructions of the enhanced subtask, and the input parameters of the enhanced subtask are input into the enhanced model.

[0156] The input parameters of the augmentation subtask are extracted by the augmentation model to obtain input feature data. Based on the input feature data, feature retrieval is performed in the knowledge base to obtain candidate feature augmentation data.

[0157] Similarity is calculated for the input parameters of the candidate feature enhancement data and the enhancement subtask to obtain the feature enhancement similarity. Candidate feature enhancement data with a feature enhancement similarity greater than the similarity threshold are identified as feature enhancement data.

[0158] The input parameters of the augmentation subtask are augmented based on the feature augmentation data to obtain the output result of the augmentation subtask; the augmented storyboard description text and augmented source images in the output result of the augmentation subtask are used as input parameters for the integration subtask.

[0159] In one possible implementation, the N subtasks further include an evaluation subtask; the model call interface information corresponding to the evaluation subtask is the sixth call interface information; the proxy component processing module is also used to perform the following operations:

[0160] If subtask S i To evaluate the subtask, the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask are determined as the input parameters for the evaluation subtask. The execution order of the integration subtask is after the execution order of the evaluation subtask, and the execution order of the evaluation subtask is after the execution order of the image generation subtask.

[0161] The input parameters of the evaluation subtask and the information of the sixth calling interface are encapsulated into the execution instructions of the evaluation subtask;

[0162] The model processing module is used to select the corresponding processing model from the model library based on the model call interface information according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0163] If subtask S i To evaluate the subtask, the evaluation model corresponding to the sixth call interface information is obtained from the model library through the execution instructions of the evaluation subtask, and the input parameters of the evaluation subtask are input into the evaluation model.

[0164] The storyboard description text is extracted using an evaluation model to obtain its features. The source image is also extracted using the same model to obtain its features. Finally, the storyboard description text features and the source image features are fused using a multimodal feature fusion method to obtain a multimodal feature vector.

[0165] The quality score is calculated on the multimodal feature vectors to obtain the quality score result. If the quality score result is greater than or equal to the score threshold, style recognition is performed on the multimodal feature vectors to obtain the style category.

[0166] If the style category matches the media data style category indicated by the generated content requirements, then the storyboard description text and source images are determined as the output of the evaluation subtask; the storyboard description text and source images in the output of the evaluation subtask are used as input parameters for the integration subtask.

[0167] One embodiment of this application provides a computer device, including: a processor, a memory, and a network interface;

[0168] The processor is connected to a memory and a network interface. The network interface is used to provide data communication functions, and the memory is used to store computer programs. When the computer program is executed by the processor, the computer device performs the method provided in the embodiments of this application.

[0169] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0170] One embodiment of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in this application embodiment.

[0171] This embodiment of the application is executed by a business server containing a task partitioning proxy component and an execution proxy component. The task partitioning proxy component partitions the generated requirements of the object input into tasks, resulting in N subtasks with an execution order. The execution proxy component then retrieves the unexecuted subtask S from the N subtasks according to the execution order. i Then, the subtask S can be analyzed using a large language model. i Model selection and recognition are performed to obtain subtask S. i The corresponding model call interface information, thereby determining the subtask S i The corresponding input parameters and execution instructions generated using the model name allow the model library to call the corresponding processing model based on the model call interface information to handle subtask S. i The input parameters are processed to obtain subtask S. i The output result when subtask S i For the Nth subtask, its output can be defined as the media data including multiple storyboard scenes as indicated by the generation requirements. The task partitioning proxy component transforms the complex generation requirements into subtasks that can be processed by the processing model, reducing the difficulty of processing the generation requirements. The execution proxy component adaptively identifies the processing model required for each subtask and schedules the corresponding processing model sequentially according to the execution order of each subtask, ensuring the rationality of data flow and the quality of the output of each subtask, thereby guaranteeing that the final generated media data is accurate and of high quality. Since the task partitioning proxy component and the execution proxy component can automatically generate media data based on the generation requirements, users only need to provide the generation requirements to efficiently and cost-effectively generate matching media data, improving the efficiency of media production. Attached Figure Description

[0172] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0173] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0174] Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0175] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 ;

[0176] Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 ;

[0177] Figure 5 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 3 ;

[0178] Figure 6 This is a schematic diagram of a system structure for generating media data provided in an embodiment of this application;

[0179] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0180] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0181] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0182] It is understood that, in the specific embodiments of this application, the user data involved requires user permission or consent when the following embodiments of this application are applied to specific products or technologies, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions.

[0183] If this application requires the collection of object data (such as user data), a prompt interface or pop-up window will be displayed before and during the collection process. This prompt interface or pop-up window is used to inform the user that certain data is being collected. The data acquisition steps will only begin after the user confirms the prompt interface or pop-up window; otherwise, the process will end. Furthermore, the acquired user data will be used in reasonable and legal scenarios or for legitimate purposes. Optionally, in scenarios where user data needs to be used but user authorization has not been obtained, authorization can be requested from the user, and the user data can only be used after authorization is granted.

[0184] This application relates to artificial intelligence technology, which will be specifically illustrated through the following embodiments.

[0185] Large Language Models (LLMs) are computer models capable of processing and generating natural language. They represent a significant advancement in artificial intelligence and hold the promise of transforming the field through learned knowledge. LLMs can predict the next word or sentence by learning statistical patterns and semantic information from language data. Their capabilities increase as the input dataset and parameter space expand. They are used in various application areas, such as machine learning and image processing, and are therefore also known as multimodal large language models.

[0186] The diffusion model is a method inspired by the physical process of gas diffusion. It has two main process domains: forward diffusion and backward diffusion. In the forward diffusion stage, the image is gradually contaminated with noise until it becomes completely random noise. In the backward diffusion process, a series of Markov chains are used to progressively remove the predicted noise at each time step, thereby recovering the data from Gaussian noise. The core of the entire model is a UNet (a convolutional network for image segmentation) network structure. Through training, it learns to predict the added noise after inputting a noisy image. Thus, when a blurred image is input, the UNet removes the predicted noise, allowing for a more accurate reconstruction of the original image.

[0187] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. For example... Figure 1As shown, the network structure may include a service server 100 and a terminal device 200. The service server 100 may include multiple proxy components. The service server 100 may have a communication connection with the terminal device 200. This communication connection is not limited to any particular method; it can be established directly or indirectly via wired communication, wireless communication, or other methods. This application does not impose any restrictions on these methods.

[0188] It should be understood that, such as Figure 1 The terminal device 200 shown can have a service application installed. When the service application runs on the terminal device 200, it can interact with the aforementioned... Figure 1 The business servers 100 shown interact with each other, enabling each business server 100 to receive dialogue requests from the terminal device 200. This business application can be a client application with image, video, or other data information capabilities, such as a game application, video editing application, social application, instant messaging application, live streaming application, short video application, video application, music application, shopping application, novel application, payment application, or browser. This application client can be a standalone client or an embedded sub-client integrated into another client (such as an instant messaging client, social client, or video client); this is not limited here.

[0189] like Figure 1 As shown, the terminal device 200 can send the generated content input by the object to the business server 100. The business server 100 may contain multiple proxy components such as a task partitioning proxy component and an execution proxy component. Through the task partitioning proxy component and the execution proxy component, media data including multiple scene contents can be generated for the generated content. The implementation process of the task partitioning proxy component and the execution proxy component can be found in the following. Figure 2 Corresponding implementation examples.

[0190] The business server 100 can send media data to the terminal device 200, and the object can view the media data through the terminal device 200. The media data can contain a combination of various data information types, such as text data, image data, etc.

[0191] Please see Figure 2 , Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application. For example... Figure 2As shown, the business server 100 can obtain the generation request content sent by the terminal device 200. The generation request content can be a prompt for a short story provided by the object, such as "Help generate a funny comic about surgery, involving dialogue between two people, 2 pages in total." It should be understood that the generation request content has no fixed format requirements and is not limited in the number of characters or the length of the comic.

[0192] The business server 100 can contain multiple proxy components, such as a task partitioning proxy component and an execution proxy component, built based on a large language model, and these proxy components communicate with each other. The task partitioning proxy component can divide the generated content into N subtasks (N is a positive integer) with an execution order, and can also submit these N subtasks to the execution proxy component. These N subtasks can include story generation subtasks, storyboard generation subtasks, image generation subtasks, and integration subtasks. The execution order of the storyboard generation subtask is after that of the story generation subtask, the image generation subtask is after that of the storyboard generation subtask, and the integration subtask is after that of the image generation subtask. Therefore, when the execution proxy component detects that none of the four subtasks have been executed, it can obtain the story generation subtask with the highest execution priority according to the execution order. This obtained story generation subtask can be called the unexecuted subtask S. i (Specifically, it can be subtask S1). If the story generation subtask has been completed, the story generation subtask with the earlier execution order can be obtained from the unexecuted storyboard generation subtask, image generation subtask, and integration subtask. The storyboard generation subtask obtained at this time can also be called the unexecuted subtask S. i+1 (Specifically, it can be a subtask S2), where i is a positive integer less than or equal to N.

[0193] Business server 100 can execute subtasks S by executing the large language model in the proxy component. i Perform model invocation and recognition to obtain subtask S i The corresponding model call interface information, such as performing model call identification on the story generation subtask to obtain the first model call interface information corresponding to the story generation subtask, that is, the model call interface information for the story generation model.

[0194] Business server 100 can be based on subtask S i The corresponding input parameters and model API information are used to generate execution instructions. Based on the execution instructions, the corresponding processing model in the model library is called. The processing model processes the input parameters to obtain subtask S. i The output of subtask S iThe corresponding input parameters are determined based on the output results or generation requirements of one or more executed subtasks. For example, the input parameters of the story generation subtask can be determined based on the generation requirements (such as prompts to indicate the generation of story text). Then, based on the input parameters and the model call interface information of the story generation model, execution instructions are generated. Finally, based on the execution instructions, the story generation model indicated by the model call interface information is called from the model library to process the input parameters, obtaining the output result of the story generation subtask, i.e., the story text content. For example, generating story text about surgery: "Little X, don't be nervous, it's just a minor surgery; Hmm? Doctor, my name isn't Little X; I know, my name is Little X."

[0195] After obtaining the output of the story generation subtask by executing the proxy component, it can be determined that the story generation subtask has been completed. Then, following the execution order, the storyboard generation subtask is obtained from the remaining unexecuted subtasks. The output of the story generation subtask, "story text content," can be input into the storyboard generation prompt in the storyboard generation subtask to obtain the updated storyboard generation prompt. This updated prompt can then be used as the input parameter for the storyboard generation subtask. The updated prompt (i.e., the input parameter) and the corresponding model call interface information for the storyboard generation subtask are encapsulated into an execution instruction for the storyboard generation subtask. The corresponding processing model in the model library is then called according to this execution instruction to process the input parameter of the storyboard generation subtask, obtaining the output of the storyboard generation subtask (such as storyboard description text). This process continues until the Nth subtask, i.e., the integration subtask, is completed. The business server 100 can then determine the output of the integration subtask as the media data including multiple storyboard contents indicated by the required content. The media data is obtained by integrating the storyboard description text from the storyboard generation subtask and the source images from the image generation subtask (the source images are obtained by the processing model corresponding to the image generation subtask, which performs text-to-image processing on the descriptions related to the main object and the background in the storyboard description text). This media data can be obtained as follows: Figure 2 The content is shown in the two pages of the comic.

[0196] The business server 100 can deliver media data to the community platform that includes community channels. Users in the community channels can browse, discuss, and share the media data, which can speed up the cold start of the channel content. The business server 100 can also receive feedback on the media data (such as likes, positive reviews, negative reviews, etc.) provided by users through the terminal device 200, and provide reference for the media data to be generated later based on the existing high-quality content in the community channels (such as channel content with high likes). In the end, high-quality media data is transmitted to the community channels, increasing more community activities and discussions, thereby improving the positive atmosphere of the community platform and the user experience.

[0197] This application embodiment uses a task partitioning proxy component to transform complex generation requirements into subtasks that can be processed by the processing model, reducing the difficulty for the model to process the generation requirements. The execution proxy component can adaptively identify the processing model required for each subtask and sequentially schedule the corresponding processing models to process each subtask according to their execution order, ensuring the rationality of data flow and the quality of the output results of each subtask. This, in turn, guarantees that the final generated media data is accurate and of high quality. Since the task partitioning proxy component and the execution proxy component can automatically generate media data based on the generation requirements, users only need to provide the generation requirements to efficiently and cost-effectively generate matching media data, improving the efficiency of media data production.

[0198] Please see Figure 3 , Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 This data processing method can be executed by a computer device, which can be, for example, Figure 1 The business server 100 shown may include a task partitioning agent component and an execution agent component; the following description will use the execution of this data processing method by the computer device as an example. This data processing method may include at least the following steps S101-S104:

[0199] Step S101: Obtain the generation requirement content input by the object; divide the generation requirement content into tasks through the task partitioning proxy component to obtain N sub-tasks with execution order, and submit the N sub-tasks to the execution proxy component; N is a positive integer;

[0200] Specifically, the computer device can obtain the generation requirements input by the object, such as a brief story content prompt: "Help generate a funny comic about xxx, involving xx people in dialogue, with xx pages in total." The generation requirements are then divided into tasks by a task partitioning proxy component. For example, the generation requirements are first divided into task intents, resulting in N task intents (N being a positive integer) with an execution order. Then, N sub-tasks are generated based on these N task intents. For instance, the N task intents may include a story generation intent. The computer device can then construct a story generation prompt for the story generation intent: "Generate a funny story script about xxx, involving a dialogue between two people," thereby generating a story generation sub-task containing the prompt: "Generate the story script content according to the prompt 'Generate a funny story script about xxx, involving a dialogue between two people'." The N task intentions can also include storyboard generation intentions. By constructing storyboard generation prompts for the storyboard generation intentions and determining the first dependency relationship between the input parameters in the storyboard generation prompts and the story generation subtasks (the input parameters of the storyboard generation subtasks can be determined based on the output results of the story generation subtasks), storyboard generation subtasks containing storyboard generation prompts and the first dependency relationship can be generated. The execution order of the storyboard generation subtasks can be determined by the first dependency relationship.

[0201] Computer devices can establish a secure communication connection between the task partitioning agent component and the execution agent component based on a security protocol (such as TLS (Transport Layer Security)). This allows the task partitioning agent component to submit N subtasks with an execution order to the execution agent component.

[0202] Step S102: In the execution proxy component, retrieve the unexecuted subtasks S from the N subtasks according to the execution order. i , where i is a positive integer less than or equal to N; subtask S is performed using a large language model. i Perform model invocation and recognition to obtain subtask S i Corresponding model call interface information;

[0203] Specifically, the computer device can retrieve the unexecuted subtask S from N subtasks in the execution agent component according to the execution order. i The N subtasks with an execution order can include story generation subtasks, storyboard generation subtasks, image generation subtasks, and integration subtasks. For example, if all N subtasks are in an unexecuted state, the story generation subtask with the highest execution priority can be designated as subtask S. iIf the story generation subtask has returned an output (i.e., the story text content), then the story generation subtask is considered executed. Further, among the subtasks that are not yet executed, the subtask with the highest execution order is selected, i.e., the storyboard generation subtask. This storyboard generation subtask is then designated as subtask S. i .

[0204] After obtaining the unexecuted subtask S i Then, the subtask S can be analyzed using a large language model. i Model call identification is performed, that is, through subtask S i The task intent is to identify the first data type of the required input parameters and the second data type of the expected output result, and then, within the large language model, to classify the subtask S according to the first and second data types. i Perform model call identification to identify those capable of accurately executing subtask S. i The processing model is identified by recognizing the corresponding model call interface information. Here, the model call interface information refers to the call interface used to invoke the identified processing model in the model library.

[0205] For example, if the subtask S is not executed i For the sub-task of storyboard generation, the computer device can identify the first data type (text type) of the input parameters and the second data type (text type) of the expected output result (such as the storyboard description text) based on the storyboard generation intention. Then, the large language model can perform model call recognition on the sub-task of storyboard generation based on the input text type and the expected output text type to obtain the model call interface information of the sub-task of storyboard generation, that is, the call interface information of the storyboard generation model (large language model).

[0206] Step S103, according to subtask S i The corresponding input parameters and model API information are used to generate execution instructions. Based on the execution instructions, the corresponding processing model in the model library is called. The processing model processes the input parameters to obtain subtask S. i The output result;

[0207] Specifically, computer devices can be configured according to subtasks S i The corresponding input parameters and model call interface information generate execution instructions, where subtask S i The corresponding input parameters are determined based on the output results or generated requirements of one or more executed subtasks. When subtask S i When executing the highest-priority subtask, a subtask S can be generated based on the generated requirements. i The corresponding input parameters, and based on subtask S iThe corresponding input parameters and model call interface information are used to generate execution instructions; when subtask S i If the subtask is not the highest priority subtask in the execution order, then the subtask S is determined based on the output results of one or more already executed subtasks. i The corresponding input parameters, and based on subtask S i Based on the corresponding input parameters and model call interface information, an execution instruction is generated; one or more executed subtasks include subtask S. i-1 (i.e., the execution order is in subtask S) i The input parameters of the previous subtask, such as the story generation subtask, are determined by the content to be generated, while the input parameters of the storyboard generation subtask are determined by the story text content output by the already executed story generation subtask.

[0208] Computer equipment can call the processing model corresponding to the model call interface information in the model library according to the execution instructions, and process the input parameters through the processing model to obtain the subtask S. i The output result.

[0209] Taking N subtasks, including story generation, storyboard generation, image generation, and integration, as an example, since the story generation subtask has the highest execution priority, the story generation subtask (i.e., subtask S1) is obtained first. The input parameters of the story generation subtask are generated based on the generation requirements. The input data in the story generation subtask and the first calling interface information (the model calling interface information corresponding to the story generation subtask) are encapsulated into the execution instruction corresponding to the story generation subtask. The processing model indicated by the first calling interface information is called in the model library through this execution instruction to process the input parameters of the story generation subtask and obtain the output result of the story generation subtask.

[0210] After receiving the output of the story generation subtask, the execution agent component can determine that the story generation subtask has been completed. Then, according to the execution order, it obtains the unexecuted storyboard generation subtask (i.e., subtask S2). It inputs the story text content from the output of the story generation subtask into the storyboard generation prompt in the storyboard generation subtask to obtain the updated storyboard generation prompt. The updated storyboard generation prompt (i.e., the input parameter of the storyboard generation subtask) and the second call interface information (the model call interface information corresponding to the storyboard generation subtask) are encapsulated into the execution instruction corresponding to the storyboard generation subtask. Then, according to the execution instruction, the processing model indicated by the second call interface information is called in the model library to process the input parameters of the storyboard generation subtask and obtain the output of the storyboard generation subtask.

[0211] After receiving the output of the storyboard generation subtask, the execution agent component can determine that the storyboard generation subtask has been completed. Then, according to the execution order, it retrieves the unexecuted image generation subtask (i.e., subtask S3). Based on the storyboard description text in the output of the storyboard generation subtask, it generates the input parameters for the image generation subtask. The input parameters and the third calling interface information (the model calling interface information corresponding to the image generation subtask) are encapsulated into an execution instruction corresponding to the image generation subtask. Then, in the model library, the processing model indicated by the third calling interface information is called according to the execution instruction to process the input parameters of the image generation subtask, obtaining the output of the image generation subtask. The output of the image generation subtask may include source images, which are obtained by the processing model corresponding to the image generation subtask through text-to-image processing of the descriptions related to the main object and the background in the storyboard description text.

[0212] After receiving the output of the image generation subtask, the execution agent component can determine that the image generation subtask has been completed. Then, according to the execution order, it obtains the unexecuted integration subtask (i.e., subtask S4). Based on the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask, it generates the input parameters of the integration subtask. The input parameters of the integration subtask and the fourth calling interface information (the model calling interface information corresponding to the integration subtask) are encapsulated into the execution instruction corresponding to the integration subtask. Then, according to the execution instruction, the processing model indicated by the fourth calling interface information is called in the model library to process the input parameters of the integration subtask, and the output result of the integration subtask is obtained, that is, the output result of the Nth subtask.

[0213] Step S104, when subtask S i When it is the Nth subtask, the output of the Nth subtask is determined as the media data including multiple storyboard content indicated by the generated content requirements.

[0214] Specifically, the input parameters of the Nth subtask are determined by the output results of one or more executed subtasks from the previous N-1 subtasks, and the output result of the Nth subtask is obtained by integrating the source data (which may include the storyboard description text and source images mentioned above) from the output results of the previous N-1 subtasks. When subtask S i When it is the Nth subtask, the computer device can determine the output of the Nth subtask as the media data including multiple storyboards indicated by the generated content requirements, and then deliver the media data to the platform.

[0215] This embodiment of the application is executed by a business server containing a task partitioning proxy component and an execution proxy component. The task partitioning proxy component partitions the generated requirements of the object input into tasks, resulting in N subtasks with an execution order. The execution proxy component then retrieves the unexecuted subtask S from the N subtasks according to the execution order. i Then, the subtask S can be analyzed using a large language model. i Model selection and recognition are performed to obtain subtask S. i The corresponding model call interface information, thereby determining the subtask S i The corresponding input parameters and execution instructions generated using the model name allow the model library to call the corresponding processing model based on the model call interface information to handle subtask S. i The input parameters are processed to obtain subtask S. i The output result when subtask S i For the Nth subtask, its output can be defined as the media data including multiple storyboard scenes as indicated by the generation requirements. The task partitioning proxy component transforms the complex generation requirements into subtasks that can be processed by the processing model, reducing the difficulty of processing the generation requirements. The execution proxy component adaptively identifies the processing model required for each subtask and schedules the corresponding processing model sequentially according to the execution order of each subtask, ensuring the rationality of data flow and the quality of the output of each subtask, thereby guaranteeing that the final generated media data is accurate and of high quality. Since the task partitioning proxy component and the execution proxy component can automatically generate media data based on the generation requirements, users only need to provide the generation requirements to efficiently and cost-effectively generate matching media data, improving the efficiency of media production.

[0216] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 This data processing method can be executed by a computer device, which can be, for example, Figure 1 The business server 100 shown may include a task partitioning agent component and an execution agent component; the following description will use the execution of this data processing method by the computer device as an example. This data processing method may include at least the following steps S201-S207:

[0217] Step S201: Divide the generated requirements into tasks using the task partitioning proxy component to obtain N sub-tasks with an execution order, and submit the N sub-tasks to the execution proxy component.

[0218] For details, please refer to the following: Figure 6 , Figure 6 This is a schematic diagram of a system structure for generating media data provided in an embodiment of this application, such as... Figure 6 As shown, the computer device uses a task partitioning agent component to divide the input generation requirements into N subtasks with an execution order (i.e., ...). Figure 6 The process of formulating the "solution" in the text can be:

[0219] The task partitioning proxy component identifies the task intent of the generated content, resulting in N task intents. The story generation intent is obtained from these N task intents, and story generation prompts are constructed for it, generating a story generation subtask containing these prompts. The storyboard generation intent is also obtained from these N task intents, and storyboard generation prompts are constructed for it. The first dependency relationship between the input parameters in the storyboard generation prompts and the story generation subtask is determined, generating a storyboard generation subtask containing both the prompts and the first dependency relationship. The image generation intent is obtained from these N task intents, and the second dependency relationship between the input parameters required for the image generation intent and the storyboard generation subtask is determined, generating an image generation subtask containing both the intent and the second dependency relationship. The integration intent is obtained from these N task intents, and the third dependency relationship between the input parameters required for the integration intent and both the storyboard and image generation subtasks is determined, generating an integration subtask containing both the integration intent and the third dependency relationship. Based on the first, second, and third dependencies, the story generation subtask, storyboard generation subtask, image generation subtask, and integration subtask are identified as N subtasks with an execution order.

[0220] For example, the input requirement could be "Help generate a funny comic about surgery, involving a dialogue between two people, for a total of 2 pages." The computer device could then construct story generation prompts for the corresponding story generation intent: "Generate a funny story script about surgery, involving a dialogue between two people." This would generate a story generation subtask containing the story generation prompts: "Generate the story script content according to the story generation prompts 'Generate a funny story script about surgery, involving a dialogue between two people'."

[0221] Computer equipment can generate storyboard generation prompts for the required content, such as: "Given a humorous dialogue '\n{story text content}\n', you need to first analyze the humor (note the importance of highlighting humorous elements in subsequent design), then design the character image (including identity, appearance, and clothing description), and finally design the storyboard. Each storyboard includes a background description and a foreground character description (including dialogue, actions, and expressions). ### Output format must meet the following requirements: Humor: <Analysis>; Character Image: 1. <Character a>: <Description of identity, appearance, and clothing> 2. <Character b>: <Same description as above>; Storyboard 1: Background: <Detailed and objective description of the background image>; Foreground Character: 1. <Character a>: Dialogue: "<Character Dialogue>"; Actions and Expressions: <Simple actions and expressions> 1. **Character b:** Same as above; **Scene 2:** Same as above; **###Notes###:** 1. If the background is the same as the previous scene x, fill in "(Same as scene x)" for <Background Scene Description>. If the character has no lines, fill in "(No lines)". 2. Be sure to minimize the number of scenes. For example, a question and an answer should be placed in the same scene instead of splitting them into two. Also, determine the first dependency relationship between the input parameters in the scene generation prompt and the story generation subtask (i.e., the input parameters are determined by the story text content in the output of the story generation subtask). This will generate a scene generation subtask containing the scene generation prompt and the first dependency relationship: "Design scene content for the story text and process it according to the scene generation prompt to generate scene description text."

[0222] The computer device can determine the second dependency between the input parameters required to generate the image generation intent corresponding to the required content and the storyboard generation subtask (i.e., the input parameters of the image generation intent are determined by the output result (text content) of the storyboard generation subtask), thereby generating an image generation subtask containing the image generation intent and the second dependency: "Generate background image and subject image based on the text content output by the storyboard generation subtask".

[0223] Optionally, the N subtasks can also include enhancement subtasks and evaluation subtasks. That is, when generating the task intent, the computer device can generate enhancement intents and evaluation intents. The computer device can then determine the enhancement dependencies between the input parameters required to generate the enhancement intent corresponding to the generated content and the storyboard generation subtask and image generation subtask (i.e., the input parameters of the enhancement intent are determined by the storyboard description text output by the storyboard generation subtask and the source images output by the image generation subtask). This generates an enhancement subtask containing the enhancement intent and enhancement dependencies: "Retrieve and data-enhance storyboard dialogue, main images, and background images." The computer device can generate the evaluation subtask corresponding to the evaluation intent through the above-mentioned methods of generating story generation subtasks, ..., enhancement subtasks: "Perform quality scoring and classification filtering on the storyboard dialogue, main images, and background images obtained through retrieval and data enhancement (i.e., filter out storyboard dialogue, main images, and background images belonging to the comedic type)", and the integration subtask corresponding to the integration intent: "Integrate the filtered storyboard dialogue, main images, and background images to generate a comic." The story generation subtask, storyboard generation subtask, and image generation subtask are used for source material generation. Therefore, the computer device can, based on the dependencies between the subtasks, sequentially determine the story generation subtask, storyboard generation subtask, image generation subtask, enhancement subtask, evaluation subtask, and integration subtask as N subtasks with an execution order. Figure 6 The "plan" in the text.

[0224] Step S202: In the execution agent component, obtain the story generation subtask from the N subtasks according to the execution order; generate execution instructions based on the input parameters and model call interface information corresponding to the story generation subtask;

[0225] Specifically, the execution agent component is configured with a task pipeline. The computer device retrieves unexecuted subtasks S from N subtasks in the execution agent component according to the execution order. i The process (where i is a positive integer less than or equal to N) can be as follows: When it is determined through the task pipeline that none of the N subtasks have been executed, obtain the subtask S with the highest execution priority among the N subtasks through the task pipeline. i When subtask S is obtained through the task pipeline i-1 When returning the output, the result is obtained through the task pipeline from subtask S. i-1 The execution order of adjacent and unexecuted subtasks S iThe task pipeline is a system architecture that executes multiple subtasks sequentially, widely used in data processing, task scheduling, and workflow management. It improves efficiency and maintainability by defining task interfaces, implementing specific tasks, and building a pipeline framework to chain tasks together for sequential execution. The task pipeline in the execution agent component can be: Story Generation Subtask (Story Content Generation) – Storyboard Generation Subtask (Storyboard Design: Background + Task Objects and other local elements + Dialogue Text Boxes) – Image Generation Subtask (Generate Main Image, Background Image) – Enhancement Subtask (Retrieval, Data Augmentation) – Evaluation Subtask (Quality Scoring, Classification and Filtering) – Integration Subtask (Local Modification, Element Assembly).

[0226] If a computer device obtains N subtasks that are all in an unexecuted state through the task pipeline, it can obtain the story generation subtask with the highest execution priority based on the above execution order.

[0227] When subtask S i When generating subtasks for a story, the computer device can identify the first calling interface information of the story generation subtask (i.e., the model calling interface information corresponding to the story generation subtask) by using the large language model in the execution agent component, based on the data type (text type) of the input parameters (story generation prompts) of the story generation subtask and the data type (text type) of the expected output result (story text content). Then, the story generation prompts and the first calling interface information in the story generation subtask are encapsulated into the execution instructions corresponding to the story generation subtask.

[0228] Among them, the computer equipment is trained to obtain the subtask S i The process of performing model-based recognition of large language models can be as follows:

[0229] Obtain the meta-task data template and action sequence data; the action sequence data includes sample input parameters and sample output results generated by the historical model based on the sample input parameters; the meta-task data template refers to the question template associated with the processing model in the model library; based on the data type of the sample input parameters, the model attributes of the historical model, and the data type of the sample output results, construct the meta-task data template into question sample data, input the question sample data into the initial large language model, and output the predicted answer data corresponding to the question sample data through the initial large language model; generate training loss values ​​based on the action sequence data and predicted answer data, and adjust the model parameters of the initial large language model based on the training loss values ​​to obtain the parameters used for subtask S. i Large language models for model call recognition.

[0230] The meta-task data template can contain complete causal knowledge of the model processing model from the model library (functional descriptions of model mechanisms, parameter rules, functional boundaries, etc. in the API (Application Programming Interface) documentation). Its main form is a question template, such as "(a) What output can be obtained by inputting 'a' into a tool?", "(b) What input is needed to obtain output 'b'?", "(c) Is input parameter 'c' valid for this tool?", and "(d) Can this tool achieve output 'd'?". Action sequence data includes sample input parameters (such as input parameter 'e') and sample output results generated by the historical model based on the sample input parameters (such as the actual output result 'f'). By constructing the meta-task data template into question sample data based on the action sequence data, we can obtain: "(a) What output can be obtained by inputting 'e' into the historical model?", "(b) What input is needed to obtain output 'f'?", "(c) Is input parameter 'e' valid for this historical model?", and "(d) Can this historical model achieve output 'f'?". Inputting the question sample data into the initial large language model allows the initial large language model to output the predicted answer data corresponding to the question sample data.

[0231] By introducing the SFT (Supervised Fine-Tuning) method, the initial large language model can acquire knowledge of the processing models in the model library from the question sample data, and validate and promote the overall task learning in action sequence data. This allows it to accurately select from a large number of overlapping and varied processing models based on the API documentation, ultimately learning to call various processing models to complete the functions required to generate media data. The computer can calculate the model loss based on the sample output results and predicted answer data in the action sequence data, obtaining the training loss value. The model parameters of the initial large language model can be adjusted based on the training loss value to obtain the parameters suitable for subtask S. i This involves using a large language model for model invocation and recognition. For example, taking stochastic gradient descent to adjust the model parameters of an initial large language model, the basic idea is to update the model parameters along the negative gradient direction of the loss function, gradually reducing the training loss value. For the model parameter θ, the method for updating the model parameters can be defined as follows: Where α is the learning rate. This is the gradient of the training loss value with respect to the model parameters. By iteratively updating the model parameters θ along the reverse direction of the gradient, the training loss value is continuously reduced. When the training loss value falls below a preset training loss threshold, the computer can determine that the model parameters θ have converged. Therefore, the model parameters θ can be used to perform training on subtask S. i Large language models for model call recognition.

[0232] Among them, the computer equipment uses a large language model to analyze subtask S i Perform model invocation and recognition to obtain subtask S i The specific process of calling the corresponding model interface information can be as follows: In the execution proxy component, through subtask S i The task intent is to identify the first data type of the required input parameters and the second data type of the expected output result; to subtask S i The corresponding first and second data types are input into the large language model, and the large language model is used to process subtask S. i Perform model call identification to identify those capable of accurately executing subtask S. i The processing model, that is, the model call interface information corresponding to the processing model is identified. Here, the model call interface information refers to the call interface used to invoke the identified processing model in the model library. Subtask S i The corresponding model call interface information indicates that the corresponding processing model supports input data of the first data type and output data of the second data type.

[0233] Step S203: According to the execution instruction, call the story generation model in the model library, process the input parameters through the story generation model, and obtain the output result of the story generation subtask;

[0234] When subtask S i When generating subtasks for a story, the computer device can send the execution instructions corresponding to the story generation subtask to the model library through the execution agent component. Then, the processing model corresponding to the first call interface information is called in the model library through the execution instructions, that is, the story generation model (large language model) is called to process the story generation prompt words and obtain the output result of the story generation subtask.

[0235] For example, keywords (such as "surgery" or "funny comic") in the story generation prompts can be used to match tags (such as "surgery" or "funny"). The story generation model then performs a similarity search on these tags in the text library. This can be done using atomic plot units (such as standard comedic scenes like "misdiagnosis") and character relationship graphs (such as doctors and patients) to select text similar to the prompts. Further text expansion is then performed to obtain the story text output from the story generation subtask: "Xiaoming, don't be nervous, it's just a minor surgery; Hmm? Doctor, my name isn't Xiaoming; I know, my name is Xiaoming." The characters involved in the story text can be categorized into six types: "boy," "girl," "adult male," "adult female," "elderly male," and "elderly female," facilitating subsequent character illustrations. The computer device can then use the story generation model to return the output of the story generation subtask to the execution agent component.

[0236] Step S204: In the execution agent component, obtain the storyboard generation subtask from the N subtasks according to the execution order; generate execution instructions based on the input parameters and model call interface information corresponding to the storyboard generation subtask;

[0237] Specifically, the computer device can obtain the output results returned by the story generation subtask through the task pipeline, thereby determining that the story generation subtask is in an executed state. It can then further obtain the storyboard generation subtask that is adjacent to the execution order of the story generation subtask but has not been executed through the task pipeline.

[0238] When subtask S i When generating subtasks for storyboard creation, if the execution order of the subtasks for story creation follows that of the subtasks for story creation, the computer device can input the story text content from the output of the story creation subtask into the storyboard creation prompts in the subtasks for storyboard creation, thus obtaining updated storyboard creation prompts, which are the input parameters of the subtasks for storyboard creation. By executing the large language model in the proxy component, based on the data type (text type) of the input parameters and the data type (text type) of the expected output (storyboard description text) of the subtasks for storyboard creation, the second calling interface information of the subtasks for storyboard creation (i.e., the model calling interface information corresponding to the subtasks for storyboard creation) can be identified. Then, the updated storyboard creation prompts and the second calling interface information are encapsulated into the execution instructions corresponding to the subtasks for storyboard creation.

[0239] Step S205: According to the execution instruction, call the storyboard generation model in the model library, process the input parameters through the storyboard generation model, and obtain the output result of the storyboard generation subtask;

[0240] Specifically, when subtask S iWhen generating a storyboard subtask, the computer device can input the story text from the output of the story generation subtask into the storyboard generation prompts in the storyboard generation subtask, resulting in updated storyboard generation prompts: "Here's a humorous dialogue: 'Xiaoming, don't be nervous, it's just a minor surgery; Hmm? Doctor, my name isn't Xiaoming; I know, my name is Xiaoming}'. You need to first analyze the humor (pay attention to highlighting the humorous elements in subsequent designs), then design the character image (including identity, appearance, and clothing descriptions), and finally design the storyboard. Each storyboard includes a background description and a foreground character description (including dialogue, actions, and expressions). ### Output format must meet: Humor: <segment Analysis; Character Portrayal: 1. <Character a>: <Describe identity, appearance, and clothing> 2. <Character b>: <Same description as above>; Storyboard 1: Background: <Detailed and objective description of the background scene>; Foreground Characters: 1. <Character a>: Dialogue: "<Character Dialogue>"; Actions and Expressions: <Simple actions and expressions>; 2. <Character b>: <Same as above>; Storyboard 2: <Same as above>; ###Notes###: 1. If the background is the same as the previous storyboard x, fill in (same as storyboard x) for <Background Scene Description>. If the character has no dialogue, fill in (No Dialogue). 2. Be sure to keep the number of storyboards as concise as possible. For example, a question and an answer should be placed in the same storyboard, rather than split into two.

[0241] The computer device can send the execution instructions corresponding to the storyboard generation subtask to the model library through the execution agent component. Then, through the execution instructions, the processing model corresponding to the second call interface information in the model library is called, that is, the storyboard generation model (large language model) is called to process the updated storyboard generation prompt words, and the output result of the storyboard generation subtask is obtained. The specific process can be as follows:

[0242] The story text in the updated storyboard generation prompts is broken down using a storyboard generation model to obtain multiple storyboards. The main subject for dialogue in each storyboard is identified, and a storyboard object description text is generated for each storyboard based on this main subject. Storyboard dialogue text is also generated for each storyboard based on this storyboard. The storyboard generation model performs style analysis on the story text to obtain style analysis information. Background description text for each storyboard is generated based on this style analysis information. The storyboard object description text, storyboard dialogue text, style analysis information, and background description text are then combined to form the output of the storyboard generation subtask. The computer device can use the storyboard generation model to return the output of the storyboard generation subtask to the execution agent component.

[0243] For example, the main subject of each split text content can be the doctor and patient in the above-mentioned updated storyboard generation prompts. The output result of the storyboard generation subtask (storyboard description text) can be: "### Output format must meet: Humor: <Name Misunderstanding>; Character Image: 1. <Character a>: <Doctor; serious but slightly tired; white medical coat> 2. <Character b>: <Patient; messy hair, pale face; blue and white striped hospital gown>; Storyboard 1: Background: <Operating room, shadowless lamps illuminating stainless steel instrument table, electrocardiogram monitor making regular beeping sounds, red "Operation in Progress" indicator light on the wall>; Foreground Characters: 1. <Character a>: Dialogue: "<Xiao Ming, don't be nervous, this is just a minor surgery>"; Actions and Expressions: <Flipping through the notebook; calm smile>; 2. <Character b>: Dialogue: "<Hmm? Doctor, my name isn't Xiaoming; Actions and expressions: <Waving hands; Confused and nervous>; Scene 2: Background: <Same as scene 1>; Foreground characters: 1. <Character a>: Dialogue: "<I know, my name is Xiaoming>"; Actions and expressions: <Pointing to the ID card; Raising eyebrows and smiling>; 2. <Character b>: Dialogue: "<No dialogue>"; Actions and expressions: <Shrugs; Bitter smile>.

[0244] Step S206: In the execution agent component, obtain the image generation subtask from the N subtasks according to the execution order; generate execution instructions based on the input parameters and model call interface information corresponding to the image generation subtask;

[0245] Specifically, the computer device can obtain the output results returned by the storyboard generation subtask through the task pipeline, thereby determining that the storyboard generation subtask is in an executed state. Furthermore, it can obtain the image generation subtask that is adjacent to the execution order of the storyboard generation subtask but has not been executed through the task pipeline.

[0246] When subtask S i When performing an image generation subtask, if the execution order of the image generation subtask follows that of the storyboard generation subtask, the computer device can use the storyboard description text as the input parameter for the image generation subtask. By executing the large language model in the proxy component, based on the data type of the input parameter (text type) and the data type of the expected output (background image, character image, etc., image type), the third calling interface information of the image generation subtask (i.e., the model calling interface information corresponding to the image generation subtask) can be identified. Then, the storyboard description text and the third calling interface information are encapsulated into the execution instructions corresponding to the image generation subtask.

[0247] Step S207: According to the execution instruction, call the image generation model in the model library, process the input parameters through the image generation model, and obtain the output result of the image generation subtask.

[0248] Specifically, when subtask Si When performing an image generation subtask, the computer device can send the execution instructions corresponding to the image generation subtask to the model library through the execution agent component. Then, the processing model corresponding to the third call interface information is called in the model library through the execution instructions. That is, the image generation model is called to process the storyboard description text in the output result of the storyboard generation subtask to obtain the output result of the image generation subtask.

[0249] The image generation model can be a diffusion model, such as a Latent Diffusion Model (LDM), a Denoising Diffusion Probabilistic Model (DDPM), etc., and this embodiment of the application is not limited thereto. The image generation model may include a text encoding layer, a forward diffusion network layer, a backward diffusion network layer, and an image decoding layer. The image decoding layer may be a decoder in a variational autoencoder, and the text encoding layer may be a text encoder, such as a pre-trained image text encoding model (Contrastive Language-Image Pre-Training, CLIP), a Bidirectional Encoder Representations from Transformers (BERT), etc.

[0250] The process of calling an image generation model to process the storyboard description text and obtain the output of the image generation subtask can be as follows: The storyboard description text in the execution instruction is input into the image generation model. The image generation model fuses and encodes the storyboard object description text and the storyboard dialogue text to obtain the main text encoding vector. The storyboard dialogue text and the storyboard background description text are also fused and encoded to obtain the background text encoding vector. A noisy image is acquired and input into the forward diffusion network layer in the image generation model. The forward diffusion network layer outputs a forward noise vector. The forward noise vector, the main text encoding vector, and the background text encoding vector are input into the backward diffusion network layer in the image generation model. In the backward diffusion network layer, the forward noise vector is denoised using the main text encoding vector to obtain a main image that matches the storyboard object description text and the storyboard dialogue text. In the backward diffusion network layer, the forward noise vector is denoised using the background text encoding vector to obtain a background image that matches the storyboard dialogue text and the storyboard background description text. The main image and the background image are then determined as the output of the image generation subtask.

[0251] Specifically, the storyboard description text in the output of the storyboard generation subtask can include storyboard object description text, storyboard dialogue text, and storyboard background description text. The computer device can use a text encoder to encode the storyboard dialogue text to obtain a dialogue text vector, to encode the storyboard object description text to obtain an object text vector, and to encode the storyboard background description text to obtain a background text vector. By concatenating or fusing the dialogue text vector and the object text vector, the main text encoding vector can be obtained; similarly, by concatenating or fusing the dialogue text vector and the background text vector, the background text encoding vector can be obtained. The computer device can add random noise vectors to the acquired noisy image at T time steps based on a forward diffusion network layer. When the noisy image approaches an isotropic Gaussian distribution N(0,1), the forward diffusion process is considered complete, and a forward noise vector is obtained. Here, T is a positive integer, the random noise vector is obtained based on random sampling, and the random noise vectors added at the T time steps are all different. The time steps indicate the magnitude of the added noise.

[0252] Computer devices can input forward noise vectors, main text encoding vectors, and background text encoding vectors into the backdiffusion network layer of an image generation model. Then, in the backdiffusion network layer, the main text encoding vector is used to denoise the forward noise vector, resulting in a main image that matches the storyboard description text and dialogue text. For example, cross-attention processing can be performed on the forward noise vector, main text encoding vector, and background text encoding vector in the backdiffusion network layer, that is, the main text encoding vector is compared with the query parameter matrix W. Q Perform a dot product operation to obtain the query vector Q; combine the main text encoding vector with the key parameter matrix W. K Perform a dot product operation to obtain the key vector K; combine the forward noise vector with the value parameter matrix W. V Performing a dot product operation yields a value vector V. The computer device can then transpose the query vector Q and the key vector K to obtain the result K. T Perform a dot product operation to obtain the attention score vector. This is based on the dimension d of the key vector K. K square root The attention score matrix is ​​dimensionality reduced, and the dimensionality-reduced attention score vector is normalized to obtain a normalized vector. The normalized vector is then multiplied by the value vector V to obtain the feature vector Z (e.g., the Tth feature vector in the T time steps). i The noise vector predicted at each time step can be obtained as shown in formula (1):

[0253]

[0254] Therefore, by adjusting the range of time step T, noise removal in specific regions can be guided by a cross-attention mechanism based on feature vectors. When 100 ≤ T < 300, global pose regions of the forward noise vector can be denoised, such as the regions of the doctor and patient figures. When 300 ≤ T < 700, detailed denoising of local regions can be performed, such as denoising and enhancing areas like the folds in the doctor's coat. When T ≥ 700, cross-modal verification and correction can be performed, such as local image correction of the ID card. After executing T time steps, the computer device can decode the noisy image based on the final noise vector predicted by the backward process (i.e., the noise vector predicted at the Tth time step) through the image decoding layer, obtaining a main image that matches the description text of the storyboard object and the dialogue text of the storyboard. The character images in the main image can serve as references for character design in subsequent storyboards, allowing for adjustments to the character images based on the actions and expressions in subsequent storyboards, while ensuring that the same character image in all storyboards has a consistent style. When choosing character IP (Intellectual Property), you can refer to the facial expressions of characters in various comedies. Using these real-life reference images can generate better (more subtle and humorous) cartoon-style illustrations.

[0255] Based on the above process, the computer device can denoise the forward noise vector using the background text encoding vector to obtain a background image that matches the storyboard dialogue text and the storyboard background description text. The main image and the background image are determined as the output results of the image generation subtask. The output results of the image generation subtask can be returned to the execution agent component through the image generation model.

[0256] Please see Figure 5 , Figure 5 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 3 This data processing method can be executed by a computer device, which can be, for example, Figure 1 The business server 100 shown may include a task partitioning agent component and an execution agent component; the following description will use the execution of this data processing method by the computer device as an example. This data processing method may include at least the following step S208 (step S208 connects to...). Figure 4 Steps S207-S214 in the corresponding embodiment:

[0257] Step S208: In the execution agent component, obtain the enhancement subtasks from the N subtasks according to the execution order; generate execution instructions based on the input parameters and model call interface information corresponding to the enhancement subtasks;

[0258] Specifically, the computer device can obtain the output results returned by the above image generation subtask through the task pipeline, thereby determining that the image generation subtask is in an executed state. Furthermore, it can obtain the enhancement subtask that is adjacent to the execution order of the image generation subtask but has not been executed through the task pipeline.

[0259] When subtask S i When enhancing a subtask, if the execution order of the enhancement subtask follows that of the image generation subtask, the computer device can use the storyboard description text from the output of the storyboard generation subtask and the source images from the output of the image generation subtask as input parameters for the enhancement subtask. By executing the large language model in the proxy component, based on the data types (text and image types) of the input parameters and the expected output data types (text and image types) of the enhancement subtask, the fifth call interface information (i.e., the model call interface information corresponding to the enhancement subtask) can be identified. Then, the input parameters and the fifth call interface information of the enhancement subtask are encapsulated into the execution instructions corresponding to the enhancement subtask.

[0260] Step S209: According to the execution instruction, call the augmented model in the model library, process the input parameters through the augmented model, and obtain the output result of the augmented subtask;

[0261] Specifically, when subtask S i When enhancing a subtask, the computer device can send the execution instructions corresponding to the enhanced subtask to the model library through the execution agent component. Then, through these execution instructions, the processing model corresponding to the fifth call interface information is invoked in the model library. That is, the enhanced model is invoked to process (retrieve enhancements) the input parameters of the enhanced subtask, obtaining the output result of the enhanced subtask. The specific process can be as follows:

[0262] The input parameters of the enhancement subtask are feature-extracted using an enhancement model to obtain input feature data. Feature retrieval is then performed in a knowledge base based on this input feature data to obtain candidate feature enhancement data. Similarity is calculated between the candidate feature enhancement data and the input parameters of the enhancement subtask to obtain feature enhancement similarity. Candidate feature enhancement data with a similarity greater than a similarity threshold are identified as feature enhancement data. Data enhancement is then applied to the input parameters of the enhancement subtask based on the feature enhancement data to obtain the output result of the enhancement subtask. The enhanced storyboard description text and enhanced source images in the output result of the enhancement subtask are used as input parameters for the integration subtask.

[0263] The enhancement model can be a Retrieval-Augmented Generation (RAG) model, which enhances the model's ability to handle knowledge-intensive tasks by retrieving relevant information from an external knowledge base. During the retrieval phase, the computer device can use the RAG model to extract features from the input parameters, obtaining input feature data. This includes character feature data (such as doctor's identity, attire, etc.) from the storyboard description text, dialogue feature data (such as dialogue style, keywords, etc.) from the dialogue text, and image features (such as image color, image style, etc.) from the background and character images in the source images. The input feature data is then used as an index in the knowledge base for feature retrieval, such as searching for rhetoric and famous quotes based on character and dialogue feature data, and unifying image style and correcting image color in the source images, resulting in candidate feature enhancement data. The computer equipment also needs to calculate the similarity between the candidate feature enhancement data and the input parameters of the enhancement subtask (e.g., cosine similarity) to obtain the feature enhancement similarity. Candidate feature enhancement data with a similarity greater than a similarity threshold (e.g., 0.8) are identified as feature enhancement data, while those with a similarity less than or equal to the threshold are deleted. During the enhancement phase, the input parameters are enhanced using the feature enhancement data, such as describing the identities of doctors and patients or adding humorous elements to their dialogue. The generation phase yields the output results of the enhancement subtask (enhanced storyboard description text and enhanced source images, etc.). The computer equipment can then return the output results of the enhancement subtask to the execution agent component through the enhancement model.

[0264] Step S210: In the execution agent component, the evaluation subtasks are obtained from the N subtasks according to the execution order; the execution instructions are generated based on the input parameters and model call interface information corresponding to the evaluation subtasks.

[0265] Specifically, the computer device can obtain the output results returned by the enhanced subtask through the task pipeline, thereby determining that the enhanced subtask is in an executed state. Furthermore, it can obtain the evaluation subtask that is adjacent to the execution order of the enhanced subtask but has not been executed through the task pipeline.

[0266] When subtask S iWhen evaluating a subtask, if the evaluation subtask is executed after the enhancement subtask, the computer device can use the (enhanced) storyboard description text from the output of the enhancement subtask and the (enhanced) source images from the output of the image generation subtask as input parameters for the evaluation subtask. By executing the large language model in the proxy component, based on the data types (text and image) of the input parameters and the expected output data types (text and image), the sixth call interface information (i.e., the model call interface information corresponding to the evaluation subtask) can be identified. Then, the input parameters and the sixth call interface information are encapsulated into the execution instructions corresponding to the evaluation subtask.

[0267] Step S211: According to the execution instruction, call the evaluation model in the model library, process the input parameters through the evaluation model, and obtain the output result of the evaluation subtask;

[0268] Specifically, when subtask S i When evaluating a subtask, the computer device can send the execution instructions corresponding to the subtask to the model library through the execution agent component. Then, the execution instructions are used to call the processing model corresponding to the sixth call interface information in the model library. That is, the evaluation model is called to perform content understanding and processing on the input parameters of the subtask, obtaining the output result of the subtask. The specific process can be as follows:

[0269] The evaluation model extracts features from the storyboard description text to obtain storyboard description text features, and also extracts features from the source images to obtain source image features. The storyboard description text can be the enhanced storyboard description text output from the enhancement subtask, and the source images can be the enhanced source images output from the enhancement subtask. The storyboard description text features and source image features are then fused using a multimodal feature fusion method to obtain a multimodal feature vector. A quality score is applied to the multimodal feature vector to obtain a quality score result. If the quality score result is greater than or equal to a score threshold, style recognition is performed on the multimodal feature vector to obtain a style category. If the style category matches the style category of the media data indicated by the generated content requirements, the storyboard description text and source images are determined as the output results of the evaluation subtask. The storyboard description text and source images in the output results of the evaluation subtask are used as input parameters for the integration subtask.

[0270] The evaluation model can be an MLLM (Multilingual Large Language Model). The computer device can extract features from the storyboard description text through the text feature encoding layer of the MLLM model to obtain the storyboard description text features, extract features from the source image through the image feature encoding layer of the MLLM model to obtain the source image features, and fuse the storyboard description text features and source image features through a multilayer perceptron or cross-attention mechanism to obtain a multimodal feature vector. Computer devices can perform quality scoring on multimodal feature vectors to obtain quality score results. For example, the CLIP model can be used to calculate the image-text matching degree of multimodal feature vectors to obtain an image-text matching score; the BERT model can be used to perform text sentiment analysis on multimodal feature vectors to obtain a text score; and the NIMA (Neural Image Assessment) model can be used to evaluate the image sharpness, composition, and other indicators of multimodal feature vectors to obtain an image score. Then, the image-text matching score, text score, and image score (all out of 10) are weighted and summed to obtain the quality score result, such as quality score result = 0.5 × image-text matching score + 0.3 × text score + 0.2 × image score.

[0271] When the quality score is greater than or equal to the score threshold (e.g., 8 points), the computer device can perform style recognition on the multimodal feature vectors to obtain style categories (e.g., humor, pathos, etc.). For example, a recognition model such as a humor detection model fine-tuned based on RoBERTa-Humor (a robust optimization of BERT pre-training method for humor) can be used to identify the text style category of the multimodal feature vectors, and the image style category (e.g., facial expressions) of the multimodal feature vectors can be identified through the CLIP model. Then, the text style category and the image style category can be classified and filtered (e.g., filtering for the funny category). If the text style category is consistent with the image style category and matches the style category of the media data indicated by the generated content requirements, then the storyboard description text and the source images can be identified as the enhanced storyboard description text and enhanced source images in the output of the evaluation subtask. The computer device can return the output of the evaluation subtask to the execution agent component through the evaluation model.

[0272] Step S212: In the execution agent component, obtain the integrated subtask from the N subtasks according to the execution order; generate the execution instruction based on the input parameters and model call interface information corresponding to the integrated subtask;

[0273] Specifically, the computer device can obtain the output results returned by the evaluation subtask through the task pipeline, thereby determining that the evaluation subtask is in an executed state. Furthermore, it can obtain the integration subtask that is adjacent to the execution order of the evaluation subtask but has not been executed through the task pipeline.

[0274] When subtask S i When integrating subtasks, if the execution order of the integrating subtasks follows that of the evaluation subtasks, the computer device can use the (enhanced) storyboard description text from the output of the evaluation subtask and the (enhanced) source images from the output of the image generation subtask as input parameters for the integrating subtask. By executing the large language model in the proxy component, based on the data types (text and image types) of the input parameters and the expected data types (text and image types) of the integrating subtask, the fourth call interface information (i.e., the model call interface information corresponding to the integrating subtask) can be identified. Then, the input parameters and the fourth call interface information of the integrating subtask are encapsulated into the execution instructions corresponding to the integrating subtask.

[0275] Step S213: According to the execution instruction, call the integrated model in the model library, process the input parameters through the integrated model, and obtain the output result of the integrated subtask;

[0276] Specifically, when subtask S i When integrating subtasks, the computer device can send the execution instructions corresponding to the integrated subtasks to the model library through the execution agent component. Then, the execution instructions are used to call the processing model corresponding to the fourth call interface information in the model library. That is, the integrated model is called to edit and process the input parameters of the integrated subtasks to obtain the output results. The specific process can be as follows:

[0277] If subtask S iTo integrate subtasks, the integration model corresponding to the fourth call interface information is selected from the model library through the execution instructions corresponding to the integration subtasks. The storyboard dialogue text, style analysis information, and the main image and background image from the source images are input parameters into the integration model. The storyboard dialogue text and style analysis information can be the enhanced storyboard dialogue text and enhanced style analysis information from the enhanced storyboard description text output by the evaluation subtask. The main image and background image can be the enhanced main image and enhanced background image from the enhanced source images output by the evaluation subtask. The integration model then crops the main image. The process involves: trimming and modifying the subject image to obtain a corrected image; redrawing and modifying the background image based on its features to obtain a corrected background image; identifying the dialogue style of the storyboard text based on style analysis information; creating dialogue text boxes with the dialogue style; filling the dialogue text boxes with the storyboard text to obtain dialogue embedding information; and then fusing the corrected subject image and dialogue embedding information into the corrected background image (e.g., stitching the corrected subject image and background image together, embedding dialogue text boxes based on the storyboard text, and adding text bubbles of a specific style to the corrected background image at appropriate positions) to obtain the output of the integration subtask.

[0278] The integrated model can include an Inpainting model and an instructionDiff model. The computer can use the Inpainting model to crop and modify the main image, such as cutting out characters in a storyboard, correcting character movements (e.g., adjusting the doctor's posture while reviewing records) using modules like OpenPose, and correcting facial expressions (e.g., enhancing the doctor's smile). This results in a corrected main image. In the instructionDiff model, based on the image features (e.g., image color) of the corrected main image, the background image can be redrawn and modified (e.g., colored background, blurring), resulting in a corrected background image. For example, based on the doctor's blue coat, the operating room environment in the background image can be redrawn by approximately 0.7 times. Furthermore, the Deep ControlNet can be used to match the doctor's image with the lighting direction of the operating room, ensuring a better fit between the doctor's image and the background image while preserving the original operating room structure. Computer devices can also identify the dialogue style of storyboard text based on style analysis information, and create dialogue text boxes with the dialogue style according to the Inpainting model. For example, when the dialogue style is passionate and energetic, the Inpaint Anything plugin can be used to create dialogue text boxes with flame or needle shapes, and then the storyboard dialogue text can be input into the dialogue text boxes to obtain dialogue embedding information. Tools such as Adobe Fresco can be used to align the subject correction image and dialogue embedding information into layers and integrate them into the background correction image to obtain the output of the integration subtask. The computer device can return the output of the integration subtask to the execution agent component through the integration model, thereby confirming that all N subtasks are in an executed state, and thus determining the output of the integration subtask as the media data including multiple storyboard content indicated by the generated content requirements. The computer device can generate several sets of media data in batches for the same theme. Manual selection is first performed from a sufficient amount of media data. Once the configuration strategy in the main path of the model is stable or the usability of the generated results reaches the business threshold, manual intervention can be removed.

[0279] Step S214: Submit media data to the analysis agent component by executing the agent component;

[0280] Specifically, computer devices can submit media data to an analysis agent component via an execution agent component. The media data includes storyboard images and dialogue text for each storyboard scene. The process of evaluating the media data and obtaining a quality score through the analysis agent component can be as follows: The analysis agent component performs dialogue style identification on the dialogue text for each storyboard scene to determine the dialogue style type; it extracts features from the storyboard images to obtain image features, and identifies the image style type of the storyboard images based on these features; it generates an image quality score based on the matching relationship between image features, dialogue style type, and image style type; it performs semantic coherence identification on the dialogue text for each storyboard scene to obtain a dialogue quality score; and it generates a quality score for the media data based on the dialogue quality score and the image quality score.

[0281] The process of generating image quality scores can be referenced in the process of generating quality assessment scores described above. Computer equipment can use the BERT model to identify and score the dialogue text in each scene, assessing aspects such as subject consistency, terminology accuracy, and clarity of reference (e.g., the accuracy of title references), thus obtaining a dialogue quality score. A quality score (out of 10) is generated by weighted summing of the dialogue quality score and the image quality score, e.g., Quality Score = 0.5 × Image / Text Quality Score + 0.5 × Dialogue Quality Score. If the quality score is greater than or equal to a threshold (e.g., 8 points), the media data is deployed to a platform. High-quality media data (e.g., high number of likes) is selected based on user browsing feedback on the media data on the platform. Alternatively, high-likes platform data with good posterior effects can be selected from existing platform data to filter the media data, resulting in high-quality media data; no restriction is imposed here. If the quality score is less than the threshold, data modification suggestions are generated for the media data, i.e., recommendations for filtering and improving the media data, thereby improving the quality of the final media data deployed to the platform. When data modification feedback indicates that the task partitioning proxy component is experiencing an anomaly in dividing the generated requirements (e.g., failure to recognize the task intent of the generated requirements, or incorrect confirmation of dependencies between subtasks), the data modification feedback is submitted to the task partitioning proxy component for re-partitioning the tasks. When data modification feedback indicates an anomaly in model selection for the model library (e.g., incorrectly selecting the diffusion model for text-to-image generation instead of the large language model for text generation), the data modification feedback is submitted to the execution proxy component for regenerating execution instructions. The execution proxy component then re-selects the model based on the execution instructions, or retrains the large language model used for model call recognition based on the data modification feedback.

[0282] This embodiment of the application is executed by a business server containing a task partitioning proxy component and an execution proxy component. The task partitioning proxy component partitions the generated requirements of the object input into tasks, resulting in N subtasks with an execution order. The execution proxy component then retrieves the unexecuted subtask S from the N subtasks according to the execution order. i Then, the subtask S can be analyzed using a large language model. i Model selection and recognition are performed to obtain subtask S. i The corresponding model call interface information, thereby determining the subtask S i The corresponding input parameters and execution instructions generated using the model name allow the model library to call the corresponding processing model based on the model call interface information to handle subtask S. i The input parameters are processed to obtain subtask S. i The output result when subtask S i When it is the Nth subtask, the output of the Nth subtask can be determined as the media data including multiple storyboard scenes indicated by the generation requirement content. The task partitioning proxy component transforms the complex generation requirement content into subtasks that can be processed by the processing model, reducing the difficulty of the model processing the generation requirement content. The execution proxy component can adaptively identify the processing model required for each subtask and can sequentially schedule the corresponding processing model to process the subtasks one by one according to the execution order between each subtask, ensuring the rationality of data flow and the quality of the output results of each subtask, thereby ensuring that the final generated media data is accurate and of high quality. Since the task partitioning proxy component and the execution proxy component can automatically generate media data according to the generation requirement content, users only need to provide the generation requirement content to efficiently and cost-effectively generate matching media data, improving the efficiency of media data production. Therefore, in the scenario of channel cold start, by distributing the media data generated in the embodiments of this application, the popularity of the cold-start channel can be increased, and the promotion of the new channel can be accelerated.

[0283] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 7 As shown, the data processing device includes a proxy component processing module 1100 and a model processing module 1200.

[0284] The proxy component processing module 1100 is used to obtain the generation requirements of the object input;

[0285] The proxy component processing module 1100 is also used to divide the generated requirement content into tasks through the task division proxy component, obtain N sub-tasks with execution order, and submit the N sub-tasks to the execution proxy component; N is a positive integer;

[0286] The proxy component processing module 1100 is also used to obtain unexecuted subtasks S from N subtasks in the execution order within the proxy component. i , where i is a positive integer less than or equal to N;

[0287] The proxy component processing module 1100 is also used to process subtasks S through a large language model. i Perform model invocation and recognition to obtain subtask S i Corresponding model call interface information;

[0288] The proxy component processing module 1100 is also used to process subtasks S. i The corresponding input parameters and model call interface information are used to generate execution instructions;

[0289] Model processing module 1200 is used to call the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain subtask S. i The output results; Subtask S i The corresponding input parameters are determined based on the output results or generated requirements of one or more executed subtasks;

[0290] The proxy component processing module 1100 is also used when subtask S i When it is the Nth subtask, the output of the Nth subtask is determined as the media data including multiple storyboard content indicated by the generated content requirements; the output of the Nth subtask is obtained by integrating the material data from the output of the previous N-1 subtasks.

[0291] In one possible implementation, the business server also includes an analysis proxy component; the proxy component processing module 1100 is further configured to perform the following operations:

[0292] The media data is submitted to the analytics proxy component by executing the proxy component;

[0293] The quality score is obtained by analyzing the media data through proxy components.

[0294] If the quality score is greater than or equal to the score threshold, the media data will be delivered to the platform.

[0295] If the quality score is less than the score threshold, data modification suggestions will be generated for the media data;

[0296] If the data modification feedback indicates that the task partitioning proxy component is abnormal in dividing the generated requirements into tasks, then the data modification feedback will be submitted to the task partitioning proxy component.

[0297] If the data modification comments indicate an anomaly in the model selection for the model library, the data modification comments will be submitted to the execution agent component.

[0298] In one possible implementation, the media data includes storyboard images and storyboard dialogue text for each storyboard content; the proxy component processing module 1100, when performing quality assessment on the media data by analyzing the proxy component and obtaining a quality score, specifically performs the following operations:

[0299] By analyzing the dialogue style of the dialogue text in each scene by the proxy component, the dialogue style type is obtained.

[0300] Feature extraction is performed on the storyboard images to obtain image features, and the image style type of the storyboard images is identified based on the image features;

[0301] An image quality score is generated based on the matching relationship between image features, dialogue style type, and image style type.

[0302] Semantic coherence is identified in the dialogue text of each scene to obtain a dialogue quality score;

[0303] A quality score for the media data is generated based on the dialogue quality score and the image quality score.

[0304] In one possible implementation, the proxy component processing module 1100 is used to divide the generated requirement content into N sub-tasks with an execution order through the task partitioning proxy component, specifically to perform the following operations:

[0305] The task partitioning proxy component identifies the task intent of the generated requirement content to obtain N task intents.

[0306] Obtain the story generation intent from N task intents, construct story generation prompts for the story generation intents, and generate story generation subtasks containing the story generation prompts;

[0307] Obtain storyboard generation intents from N task intents, construct storyboard generation prompts for the storyboard generation intents, determine the first dependency relationship between the input parameters in the storyboard generation prompts and the story generation subtasks, and generate storyboard generation subtasks containing storyboard generation prompts and the first dependency relationship;

[0308] Obtain the image generation intent from N task intents, determine the second dependency between the input parameters required by the image generation intent and the storyboard generation subtask, and generate an image generation subtask that includes the image generation intent and the second dependency.

[0309] Obtain the integration intent from N task intents, determine the third dependency relationship between the input parameters required for the integration intent and the storyboard generation subtask and the image generation subtask, and generate an integration subtask containing the integration intent and the third dependency relationship.

[0310] Based on the first dependency, second dependency, and third dependency, the story generation subtask, storyboard generation subtask, image generation subtask, and integration subtask are identified as N subtasks with an execution order.

[0311] In one possible implementation, the execution proxy component is configured with a task pipeline; the proxy component processing module 1100 is used to retrieve unexecuted subtasks S from N subtasks in the execution proxy component according to the execution order. i Specifically, it is used to perform the following operations:

[0312] When the task pipeline determines that none of the N subtasks have been executed, the task pipeline retrieves the subtask S with the highest execution priority among the N subtasks. i ,

[0313] When subtask S is obtained through the task pipeline i-1 When returning the output, the result is obtained through the task pipeline from subtask S. i-1 The execution order of adjacent and unexecuted subtasks S i ;

[0314] The proxy component processing module 1100 is used to process subtasks S i When generating execution instructions based on the corresponding input parameters and model call interface information, they are specifically used to perform the following operations:

[0315] When subtask S i When executing the highest-priority subtask, generate subtask S based on the generated requirements. i The corresponding input parameters, based on subtask S i The corresponding input parameters and model call interface information are used to generate execution instructions;

[0316] When subtask S i If the subtask is not the highest priority subtask in the execution order, the subtask S is determined based on the output results of one or more already executed subtasks. i The corresponding input parameters, based on subtask S i Based on the corresponding input parameters and model call interface information, an execution instruction is generated; one or more executed subtasks include subtask S. i-1 .

[0317] In one possible implementation, the N subtasks include a story generation subtask, a storyboard generation subtask, an image generation subtask, and an integration subtask; the proxy component processing module 1100 is used to process the subtasks S... i When generating execution instructions based on the corresponding input parameters and model call interface information, they are specifically used to perform the following operations:

[0318] If subtask S i For story generation subtasks, the story generation prompts and the first API call information in the story generation subtask are encapsulated into execution instructions; the first API call information is the model API call information corresponding to the story generation subtask.

[0319] If subtask S i For the storyboard generation subtask, the story text content from the output of the story generation subtask is input into the storyboard generation prompt words in the storyboard generation subtask to obtain the updated storyboard generation prompt words. The updated storyboard generation prompt words and the second call interface information are encapsulated into an execution instruction. The execution order of the storyboard generation subtask is after the execution order of the story generation subtask. The second call interface information is the model call interface information corresponding to the storyboard generation subtask.

[0320] If subtask S i For the image generation subtask, the storyboard description text and the third calling interface information in the output of the storyboard generation subtask are encapsulated into execution instructions; the execution order of the image generation subtask is after the execution order of the storyboard generation subtask; the third calling interface information is the model calling interface information corresponding to the image generation subtask.

[0321] If subtask S i To integrate subtasks, the input parameters of the integrated subtasks are obtained, and execution instructions are generated based on the input parameters of the integrated subtasks and the fourth calling interface information. The input parameters of the integrated subtasks are determined based on the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask. The execution order of the integrated subtasks follows the execution order of the image generation subtasks. The fourth calling interface information is the model calling interface information corresponding to the integrated subtasks.

[0322] In one possible implementation, the model processing module 1200 is used to select the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0323] If subtask S i For the sub-task of storyboard generation, the storyboard generation model corresponding to the second calling interface information is selected from the model library according to the execution instruction corresponding to the sub-task of storyboard generation;

[0324] The story text content in the updated storyboard generation prompts is broken down using a storyboard generation model to obtain the split text content corresponding to multiple storyboards.

[0325] Identify the main subject used for dialogue in each segmented text content, and generate a storyboard object description text for the main subject of each storyboard based on each segmented text content;

[0326] Generate storyboard dialogue text for each scene based on the content of each segmented text;

[0327] The story script content is analyzed for plot style using a storyboard generation model to obtain style analysis information.

[0328] Generate background description text for each scene based on the content of each segmented text;

[0329] The storyboard object description text, storyboard dialogue text, style analysis information, and storyboard background description text are determined as the output of the storyboard generation subtask.

[0330] In one possible implementation, the storyboard description text includes storyboard object description text, storyboard dialogue text, and storyboard background description text; the model processing module 1200 is used to select the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0331] If subtask S i For image generation subtasks, the image generation model corresponding to the third calling interface information is selected from the model library according to the execution instructions corresponding to the image generation subtask.

[0332] The storyboard description text in the execution instruction is input into the image generation model. The image generation model fuses and encodes the storyboard object description text and the storyboard dialogue text to obtain the main text encoding vector. The storyboard dialogue text and the storyboard background description text are fused and encoded to obtain the background text encoding vector.

[0333] The noisy image is acquired and input into the forward diffusion network layer in the image generation model. The forward diffusion network layer outputs a forward noise vector.

[0334] The forward noise vector, the main text encoding vector, and the background text encoding vector are input into the back diffusion network layer in the image generation model. In the back diffusion network layer, the forward noise vector is denoised by the main text encoding vector to obtain the main image that matches the storyboard object description text and the storyboard dialogue text.

[0335] In the back-diffusion network layer, the forward noise vector is denoised by the background text encoding vector to obtain a background image that matches the storyboard dialogue text and the storyboard background description text.

[0336] The main image and background image are determined as the output of the image generation subtask.

[0337] In one possible implementation, the storyboard description text includes storyboard dialogue text and style analysis information, and the source images include main images and background images; the model processing module 1200 is used to select the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0338] If subtask S i To integrate subtasks, the integration model corresponding to the fourth call interface information is selected from the model library through the execution instructions corresponding to the integration subtasks, and the storyboard dialogue text, style analysis information, main image and background image are input into the integration model as input parameters;

[0339] The subject image is cropped and modified by integrating the model to obtain the corrected subject image. The background image is then redrawn and modified based on the image features of the corrected subject image to obtain the corrected background image.

[0340] Based on the style analysis information, the dialogue style of the storyboard dialogue text is identified, a dialogue text box with the dialogue style is created, and the storyboard dialogue text is filled into the dialogue text box to obtain the dialogue embedding information;

[0341] The main image and the embedded dialogue information are fused into the background image to obtain the output of the integration subtask.

[0342] In one possible implementation, the proxy component processing module 1100 is used to process subtasks S through a large language model. i Model selection and recognition are performed to obtain subtask S. i When calling the corresponding model's API, it is specifically used to perform the following operations:

[0343] In the execution agent component, through subtask S i The task intent is to identify the first data type of the required input parameters and the second data type of the expected output result;

[0344] Subtask S i The corresponding first and second data types are input into the large language model, and the large language model is used to process subtask S. i Perform model invocation and recognition to obtain subtask Si Corresponding model call interface information; Subtask S i The corresponding model call interface information indicates that the corresponding processing model supports input data of the first data type and output data of the second data type.

[0345] In one possible implementation, the proxy component processing module 1100 is also used to perform the following operations:

[0346] Obtain the meta-task data template and action sequence data; the action sequence data includes sample input parameters and sample output results generated by the historical model based on the sample input parameters; the meta-task data template refers to the question template associated with the processing model in the model library;

[0347] Based on the data type of the sample input parameters, the model attributes of the historical model, and the data type of the sample output results, the meta-task data template is constructed into question sample data. The question sample data is input into the initial large language model, and the predicted answer data corresponding to the question sample data is output through the initial large language model.

[0348] Training loss values ​​are generated based on action sequence data and predicted response data. The model parameters of the initial large language model are then adjusted based on these training loss values ​​to obtain the parameters suitable for subtask S. i Large language models for model call recognition.

[0349] In one possible implementation, the N subtasks further include enhancement subtasks; the model call interface information corresponding to the enhancement subtasks is the fifth call interface information; the proxy component processing module 1100 is also used to perform the following operations:

[0350] If subtask S i To enhance the subtask, the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask are determined as the input parameters for the enhancement subtask. The integration subtask is executed after the enhancement subtask, and the enhancement subtask is executed after the image generation subtask.

[0351] The input parameters and fifth call interface information of the enhanced subtask are encapsulated into the execution instructions of the enhanced subtask;

[0352] The model processing module 1200 is used to select the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0353] If subtask S iTo enhance the subtask, the enhanced model corresponding to the fifth calling interface information is obtained from the model library through the execution instructions of the enhanced subtask, and the input parameters of the enhanced subtask are input into the enhanced model.

[0354] The input parameters of the augmentation subtask are extracted by the augmentation model to obtain input feature data. Based on the input feature data, feature retrieval is performed in the knowledge base to obtain candidate feature augmentation data.

[0355] Similarity is calculated for the input parameters of the candidate feature enhancement data and the enhancement subtask to obtain the feature enhancement similarity. Candidate feature enhancement data with a feature enhancement similarity greater than the similarity threshold are identified as feature enhancement data.

[0356] The input parameters of the augmentation subtask are augmented based on the feature augmentation data to obtain the output result of the augmentation subtask; the augmented storyboard description text and augmented source images in the output result of the augmentation subtask are used as input parameters for the integration subtask.

[0357] In one possible implementation, the N subtasks further include an evaluation subtask; the model call interface information corresponding to the evaluation subtask is the sixth call interface information; the proxy component processing module 1100 is also used to perform the following operations:

[0358] If subtask S i To evaluate the subtask, the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask are determined as the input parameters for the evaluation subtask. The execution order of the integration subtask is after the execution order of the evaluation subtask, and the execution order of the evaluation subtask is after the execution order of the image generation subtask.

[0359] The input parameters of the evaluation subtask and the information of the sixth calling interface are encapsulated into the execution instructions of the evaluation subtask;

[0360] The model processing module 1200 is used to select the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i When outputting the result, it is specifically used to perform the following operations:

[0361] If subtask S i To evaluate the subtask, the evaluation model corresponding to the sixth call interface information is obtained from the model library through the execution instructions of the evaluation subtask, and the input parameters of the evaluation subtask are input into the evaluation model.

[0362] The storyboard description text is extracted using an evaluation model to obtain its features. The source image is also extracted using the same model to obtain its features. Finally, the storyboard description text features and the source image features are fused using a multimodal feature fusion method to obtain a multimodal feature vector.

[0363] The quality score is calculated on the multimodal feature vectors to obtain the quality score result. If the quality score result is greater than or equal to the score threshold, style recognition is performed on the multimodal feature vectors to obtain the style category.

[0364] If the style category matches the media data style category indicated by the generated content requirements, then the storyboard description text and source images are determined as the output of the evaluation subtask; the storyboard description text and source images in the output of the evaluation subtask are used as input parameters for the integration subtask.

[0365] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0366] This embodiment of the application is executed by a business server containing a task partitioning proxy component and an execution proxy component. The task partitioning proxy component partitions the generated requirements of the object input into tasks, resulting in N subtasks with an execution order. The execution proxy component then retrieves the unexecuted subtask S from the N subtasks according to the execution order. i Then, the subtask S can be analyzed using a large language model. i Model selection and recognition are performed to obtain subtask S. i The corresponding model call interface information, thereby determining the subtask S i The corresponding input parameters and execution instructions generated using the model name allow the model library to call the corresponding processing model based on the model call interface information to handle subtask S. i The input parameters are processed to obtain subtask S. i The output result when subtask S iWhen it is the Nth subtask, the output of the Nth subtask can be determined as the media data including multiple storyboard scenes indicated by the generation requirement content. The task partitioning proxy component transforms the complex generation requirement content into subtasks that can be processed by the processing model, reducing the difficulty of the model processing the generation requirement content. The execution proxy component can adaptively identify the processing model required for each subtask and can sequentially schedule the corresponding processing model to process the subtasks one by one according to the execution order between each subtask, ensuring the rationality of data flow and the quality of the output results of each subtask, thereby ensuring that the final generated media data is accurate and of high quality. Since the task partitioning proxy component and the execution proxy component can automatically generate media data according to the generation requirement content, users only need to provide the generation requirement content to efficiently and cost-effectively generate matching media data, improving the efficiency of media data production. Therefore, in the scenario of channel cold start, by distributing the media data generated in the embodiments of this application, the popularity of the cold-start channel can be increased, and the promotion of the new channel can be accelerated.

[0367] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 8 As shown, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include an object interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The object interface 1003 may include a display screen and a keyboard; optionally, the object interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the processor 1001. Figure 8 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, an object interface module, and a device control application.

[0368] In such Figure 8 In the computer device 1000 shown, the network interface 1004 provides network communication elements; the object interface 1003 is mainly used to provide an input interface for objects; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0369] Obtain the generated requirements from the object input;

[0370] The generated requirements are divided into N subtasks with an execution order by a task partitioning proxy component. The N subtasks are then submitted to the execution proxy component; N is a positive integer.

[0371] In the execution proxy component, retrieve the unexecuted subtasks S from the N subtasks according to the execution order. i , where i is a positive integer less than or equal to N;

[0372] Using a large language model to analyze subtask S i Perform model invocation and recognition to obtain subtask S i Corresponding model call interface information;

[0373] According to subtask S i The corresponding input parameters and model API information are used to generate execution instructions. Based on the execution instructions, the corresponding processing model in the model library is called. The processing model processes the input parameters to obtain subtask S. i The output results; Subtask S i The corresponding input parameters are determined based on the output results or generated requirements of one or more executed subtasks;

[0374] When subtask S i When it is the Nth subtask, the output of the Nth subtask is determined as the media data including multiple storyboard content indicated by the generated content requirements; the output of the Nth subtask is obtained by integrating the material data from the output of the previous N-1 subtasks.

[0375] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figure 3 , Figure 4 and Figure 5 The description of the data processing method in any corresponding embodiment will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0376] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program. When the processor executes the computer program, it can execute the foregoing text. Figure 3 , Figure 4 and Figure 5The description of the data processing method in any corresponding embodiment is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0377] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been displayed or will be displayed.

[0378] Furthermore, it should be noted that this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the aforementioned... Figure 3 , Figure 4 and Figure 5 The method provided in any of the corresponding embodiments.

[0379] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0380] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the foregoing description as a network element. Whether these network elements are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described network elements using different methods for each specific application, but such implementation should not be considered beyond the scope of this application.

[0381] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0382] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0383] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0384] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method is executed by a business server, which includes a task partitioning proxy component and an execution proxy component; the method includes: Obtain the generated requirements from the object input; The task partitioning proxy component divides the generated requirements into N subtasks with an execution order, and submits the N subtasks to the execution proxy component; N is a positive integer. In the execution agent component, unexecuted subtasks S are obtained from the N subtasks according to the execution order. i , where i is a positive integer less than or equal to N; The subtask S is analyzed using a large language model. i Perform model invocation and recognition to obtain the subtask S i Corresponding model call interface information; According to the subtask S i The corresponding input parameters and the model call interface information generate an execution instruction. Based on the execution instruction, the processing model corresponding to the model call interface information is called from the model library. The input parameters are processed by the processing model to obtain the subtask S. i The output result; the subtask S i The corresponding input parameters are determined based on the output results of one or more executed subtasks or the generated requirements. When the subtask S i When it is the Nth subtask, the output of the Nth subtask is determined as the media data including multiple storyboard content indicated by the generated requirement content; the output of the Nth subtask is obtained by integrating the material data in the output of the previous N-1 subtasks.

2. The method according to claim 1, characterized in that, The business server also includes an analysis proxy component; the method further includes: The media data is submitted to the analysis agent component through the execution agent component; The media data is evaluated using the analysis agent component to obtain a quality score. If the quality score is greater than or equal to the score threshold, the media data will be delivered to the platform. If the quality score is less than the score threshold, then data modification suggestions are generated for the media data; If the data modification suggestion indicates that the task partitioning proxy component is abnormal in partitioning the generated requirement content into tasks, then the data modification suggestion is submitted to the task partitioning proxy component. If the data modification suggestion indicates an anomaly in the model selection for the model library, then the data modification suggestion is submitted to the execution agent component.

3. The method according to claim 2, characterized in that, The media data includes storyboard images and dialogue text for each storyboard scene; the quality assessment of the media data through the analysis agent component to obtain a quality score includes: The analysis agent component performs dialogue style recognition on the dialogue text in each scene to obtain the dialogue style type. Feature extraction is performed on the storyboard image to obtain image features, and the image style type of the storyboard image is identified based on the image features; An image quality score is generated based on the matching relationship between the image features, the dialogue style type, and the image style type. Semantic coherence recognition is performed on the dialogue text in each scene to obtain a dialogue quality score; A quality score for the media data is generated based on the dialogue quality score and the image quality score.

4. The method according to claim 1, characterized in that, The process of dividing the generated requirements content into N subtasks with an execution order using the task partitioning proxy component includes: The task partitioning agent component identifies the task intent of the generated requirement content to obtain N task intents. Obtain the story generation intent from the N task intents, construct story generation prompt words for the story generation intent, and generate a story generation subtask containing the story generation prompt words; Obtain the storyboard generation intent from the N task intents, construct storyboard generation prompts for the storyboard generation intent, determine the first dependency relationship between the input parameters in the storyboard generation prompts and the story generation subtask, and generate a storyboard generation subtask that includes the storyboard generation prompts and the first dependency relationship; Obtain the image generation intent from the N task intents, determine the second dependency relationship between the input parameters required by the image generation intent and the storyboard generation subtask, and generate an image generation subtask that includes the image generation intent and the second dependency relationship; Obtain the integration intent from the N task intents, determine the third dependency relationship between the input parameters required by the integration intent and the storyboard generation subtask and the image generation subtask, and generate an integration subtask that includes the integration intent and the third dependency relationship; Based on the first dependency, the second dependency, and the third dependency, the story generation subtask, the storyboard generation subtask, the image generation subtask, and the integration subtask are determined as N subtasks with an execution order.

5. The method according to claim 1, characterized in that, The execution agent component is configured with a task pipeline; the execution agent component retrieves unexecuted subtasks S from the N subtasks according to the execution order. i ,include: When the task pipeline determines that none of the N subtasks have been executed, the task pipeline retrieves the subtask S with the highest execution priority among the N subtasks. i , When subtask S is obtained through the task pipeline i-1 When returning the output result, the task pipeline obtains the result related to the subtask S. i-1 The subtasks S that are executed in an adjacent order and have not yet been executed i ; According to the subtask S i The corresponding input parameters and the model call interface information generate execution instructions, including: When the subtask S i When executing the subtask with the highest priority, the subtask S is generated based on the generated requirement content. i The corresponding input parameters, based on the subtask S i Based on the corresponding input parameters and the model call interface information, an execution instruction is generated; When the subtask S i If the subtask is not the highest priority subtask in the execution order, the subtask S is determined based on the output results of one or more already executed subtasks. i The corresponding input parameters, based on the subtask S i Based on the corresponding input parameters and the model call interface information, an execution instruction is generated; the one or more executed subtasks include the subtask S. i-1 .

6. The method according to claim 1, characterized in that, The N sub-tasks include a story generation sub-task, a storyboard generation sub-task, an image generation sub-task, and an integration sub-task; the process is based on the sub-task S i The corresponding input parameters and the model call interface information generate execution instructions, including: If the subtask S i For a story generation subtask, the story generation prompt and the first calling interface information in the story generation subtask are encapsulated into an execution instruction; the first calling interface information is the model calling interface information corresponding to the story generation subtask. If the subtask S i For the storyboard generation subtask, the story text content from the output of the story generation subtask is input into the storyboard generation prompt words in the storyboard generation subtask to obtain updated storyboard generation prompt words. The updated storyboard generation prompt words and the second calling interface information are encapsulated into an execution instruction. The execution order of the storyboard generation subtask is after the execution order of the story generation subtask. The second calling interface information is the model calling interface information corresponding to the storyboard generation subtask. If the subtask S i For the image generation subtask, the storyboard description text and the third calling interface information in the output of the storyboard generation subtask are encapsulated into an execution instruction; the execution order of the image generation subtask is after the execution order of the storyboard generation subtask; the third calling interface information is the model calling interface information corresponding to the image generation subtask. If the subtask S i To integrate subtasks, the input parameters of the integrated subtasks are obtained, and execution instructions are generated based on the input parameters of the integrated subtasks and the fourth calling interface information. The input parameters of the integrated subtasks are determined based on the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask. The execution order of the integrated subtasks follows the execution order of the image generation subtasks. The fourth calling interface information is the model calling interface information corresponding to the integrated subtasks.

7. The method according to claim 6, characterized in that, The step involves selecting the processing model corresponding to the model call interface information from the model library according to the execution instruction, and then processing the input parameters through the processing model to obtain the subtask S. i The output includes: If the subtask S i For a subtask of storyboard generation, the storyboard generation model corresponding to the second calling interface information is selected from the model library according to the execution instruction corresponding to the subtask of storyboard generation; The story text content in the updated storyboard generation prompt is split into multiple storyboards by using the storyboard generation model. Identify the main subject object used for dialogue in each segmented text content, and generate a storyboard object description text for the main subject object for each storyboard based on each segmented text content; Generate the scene dialogue text for each scene based on the content of each segmented text; The storyboard generation model is used to analyze the plot style of the story text to obtain style analysis information. Generate background description text for each scene based on the content of each segmented text; The storyboard object description text, the storyboard dialogue text, the style analysis information, and the storyboard background description text are determined as the output results of the storyboard generation subtask.

8. The method according to claim 6, characterized in that, The storyboard description text includes storyboard object description text, storyboard dialogue text, and storyboard background description text; the step involves selecting the processing model corresponding to the model call interface information from the model library according to the execution instruction, and processing the input parameters through the processing model to obtain the subtask S. i The output includes: If the subtask S i If the image generation subtask is a specific task, then the image generation model corresponding to the third calling interface information is selected from the model library according to the execution instruction corresponding to the image generation subtask. The storyboard description text in the execution instruction is input into the image generation model. The image generation model is used to fuse and encode the storyboard object description text and the storyboard dialogue text to obtain the main text encoding vector. The storyboard dialogue text and the storyboard background description text are fused and encoded to obtain the background text encoding vector. A noisy image is acquired and input into the forward diffusion network layer in the image generation model. The forward diffusion network layer outputs a forward noise vector. The forward noise vector, the main text encoding vector, and the background text encoding vector are input into the back diffusion network layer in the image generation model. In the back diffusion network layer, the forward noise vector is denoised by the main text encoding vector to obtain a main image that matches the storyboard object description text and the storyboard dialogue text. In the backward diffusion network layer, the forward noise vector is denoised by the background text encoding vector to obtain a background image that matches the storyboard dialogue text and the storyboard background description text. The main image and the background image are determined as the output results of the image generation subtask.

9. The method according to claim 6, characterized in that, The storyboard description text includes storyboard dialogue text and style analysis information; the source images include main images and background images; the process involves selecting the processing model corresponding to the model call interface information from the model library according to the execution instruction, and processing the input parameters through the processing model to obtain the subtask S. i The output includes: If the subtask S i For the integration subtask, the integration model corresponding to the fourth calling interface information is selected in the model library through the execution instruction corresponding to the integration subtask, and the storyboard dialogue text, the style analysis information, the main image and the background image are input as input parameters to the integration model; The subject image is cropped and modified using the integrated model to obtain a corrected subject image. The background image is then redrawn and modified based on the image features of the corrected subject image to obtain a corrected background image. The dialogue style of the storyboard dialogue text is identified based on the style parsing information, a dialogue text box with the dialogue style is created, and the storyboard dialogue text is filled into the dialogue text box to obtain the dialogue embedding information; The main body correction image and the dialogue embedding information are fused into the background correction image to obtain the output result of the integration subtask.

10. The method according to claim 1, characterized in that, The subtask S is analyzed using a large language model. i Model selection and recognition are performed to obtain the subtask S. i The corresponding model call interface information includes: In the execution agent component, through the subtask S i The task intent is to identify the first data type of the required input parameters and the second data type of the expected output result; Subtask S i The corresponding first data type and second data type are input into the large language model, and the large language model is used to process the subtask S. i Perform model invocation and recognition to obtain the subtask S i The corresponding model call interface information; the subtask S i The corresponding model call interface information indicates that the corresponding processing model supports input with data of the first data type and output with data of the second data type.

11. The method according to claim 1, characterized in that, The method further includes: Obtain meta-task data templates and action sequence data; the action sequence data includes sample input parameters and sample output results generated by historical models based on the sample input parameters; the meta-task data template refers to a question template associated with the processing model in the model library; Based on the data type of the sample input parameters, the model attributes of the historical model, and the data type of the sample output results, the meta-task data template is constructed into question sample data, the question sample data is input into the initial large language model, and the predicted answer data corresponding to the question sample data is output through the initial large language model. A training loss value is generated based on the action sequence data and the predicted answer data. The model parameters of the initial large language model are then adjusted based on the training loss value to obtain parameters suitable for the subtask S. i Large language models for model call recognition.

12. The method according to claim 6, characterized in that, The N subtasks also include enhanced subtasks; The model call interface information corresponding to the enhanced subtask is the fifth call interface information; the method further includes: If the subtask S i For the enhancement subtask, the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask are determined as the input parameters corresponding to the enhancement subtask; the execution order of the integration subtask is after the execution order of the enhancement subtask, and the execution order of the enhancement subtask is after the execution order of the image generation subtask; The input parameters of the enhanced subtask and the fifth calling interface information are encapsulated into the execution instructions of the enhanced subtask; The step involves selecting the processing model corresponding to the model call interface information from the model library according to the execution instruction, and then processing the input parameters through the processing model to obtain the subtask S. i The output includes: If the subtask S i For the enhanced subtask, the enhanced model corresponding to the fifth calling interface information is obtained from the model library through the execution instructions of the enhanced subtask, and the input parameters of the enhanced subtask are input into the enhanced model; The enhancement model extracts features from the input parameters of the enhancement sub-task to obtain input feature data. Based on the input feature data, feature retrieval is performed in the knowledge base to obtain candidate feature enhancement data. Similarity is calculated between the candidate feature enhancement data and the input parameters of the enhancement subtask to obtain feature enhancement similarity. Candidate feature enhancement data with feature enhancement similarity greater than a similarity threshold are identified as feature enhancement data. The input parameters of the enhancement subtask are augmented based on the feature enhancement data to obtain the output result of the enhancement subtask; the enhanced storyboard description text and enhanced source images in the output result of the enhancement subtask are used as input parameters for the integration subtask.

13. The method according to claim 6, characterized in that, The N subtasks further include an evaluation subtask; the model call interface information corresponding to the evaluation subtask is the sixth call interface information; the method further includes: If the subtask S i For the evaluation subtask, the storyboard description text in the output of the storyboard generation subtask and the source images in the output of the image generation subtask are determined as the input parameters corresponding to the evaluation subtask; the execution order of the integration subtask is after the execution order of the evaluation subtask, and the execution order of the evaluation subtask is after the execution order of the image generation subtask. The input parameters of the evaluation subtask and the sixth calling interface information are encapsulated into the execution instructions of the evaluation subtask; The step involves selecting the processing model corresponding to the model call interface information from the model library according to the execution instruction, and then processing the input parameters through the processing model to obtain the subtask S. i The output includes: If the subtask S i For the evaluation subtask, the evaluation model corresponding to the sixth calling interface information is obtained from the model library through the execution instructions of the evaluation subtask, and the input parameters of the evaluation subtask are input into the evaluation model; The evaluation model is used to extract features from the storyboard description text to obtain storyboard description text features. The evaluation model is also used to extract features from the source image to obtain source image features. The storyboard description text features and source image features are then fused using multimodal features to obtain a multimodal feature vector. The multimodal feature vector is quality scored to obtain a quality score result. If the quality score result is greater than or equal to a scoring threshold, style recognition is performed on the multimodal feature vector to obtain a style category. If the style category matches the media data style category indicated by the generated content requirement, then the storyboard description text and the source image are determined as the output of the evaluation subtask; the storyboard description text and the source image in the output of the evaluation subtask are used as input parameters for the integration subtask.

14. A data processing apparatus, characterized in that, include: The proxy component processing module is used to obtain the generation requirements of the object input; The proxy component processing module is further configured to divide the generated requirement content into tasks through the task division proxy component, obtain N sub-tasks with execution order, and submit the N sub-tasks to the execution proxy component; N is a positive integer; The proxy component processing module is further configured to obtain, in the execution proxy component, unexecuted subtasks S from the N subtasks according to the execution order. i , where i is a positive integer less than or equal to N; The proxy component processing module is also used to process the subtask S through a large language model. i Perform model invocation and recognition to obtain the subtask S i Corresponding model call interface information; The proxy component processing module is also used to process the subtask S. i The corresponding input parameters and the model call interface information are used to generate execution instructions; The model processing module is used to call the processing model corresponding to the model call interface information in the model library according to the execution instruction, and process the input parameters through the processing model to obtain the subtask S. i The output result; the subtask S i The corresponding input parameters are determined based on the output results of one or more executed subtasks or the generated requirements. The proxy component processing module is also used when the subtask S i When it is the Nth subtask, the output of the Nth subtask is determined as the media data including multiple storyboard content indicated by the generated requirement content; the output of the Nth subtask is obtained by integrating the material data in the output of the previous N-1 subtasks.

15. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method according to any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-13.

17. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium and adapted to be read and executed by a processor so that a computer device having the processor performs the method of any one of claims 1-13.

Images