Task processing method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides a task processing method and device, equipment, a medium and a program product, and the method comprises the steps: carrying out the task arrangement of a material segment contained in a media push material according to the input media push material and a prompt text, and obtaining a media generation task corresponding to the material segment; determining adaptive resource configuration in the resource configuration of the heterogeneous resource pool based on the task parameters of the media generation task; determining adaptive heterogeneous resources of the media generation task in the heterogeneous resources according to the load data of the heterogeneous resources under the adaptive resource configuration; and distributing the media generation tasks to the corresponding adaptive heterogeneous resources to perform media generation to obtain media generation contents, merging the media generation contents to obtain media push contents, and outputting the media push contents. By adopting the embodiment of the invention, the accuracy of the media push content can be improved.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of data processing, and particularly relates to a task processing method and device, equipment, medium and program product. BACKGROUND

[0002] With the continuous development of Internet technology and the continuous progress of social information, more and more business scenarios need to push media content for business scenarios in order to improve the popularity of the business. In this process, the media content pushed may be generated based on the related materials of the business scenario.

[0003] At present, in the process of generating media content, the resource configuration is not distinguished, and the appropriate machine resources are directly determined in all machine resources for media content generation, which may lead to low resource utilization and low accuracy of the generated media content. SUMMARY

[0004] An embodiment of the present specification aims to provide a task processing method, device, equipment, medium and program product to improve the accuracy of media push content.

[0005] To solve the above technical problems, an embodiment of the present specification is implemented as follows: In a first aspect, an embodiment of the present specification provides a task processing method, comprising: According to the input media push material and prompt text, task arrangement is performed on the material segment contained in the media push material to obtain the media generation task corresponding to the material segment; Based on the task parameters of the media generation task, the adaptive resource configuration is determined in the resource configuration of the heterogeneous resource pool; According to the load data of the heterogeneous resources under the adaptive resource configuration, the adaptive heterogeneous resources of the media generation task are determined in the heterogeneous resources; The media generation task is assigned to the corresponding adaptive heterogeneous resource for media generation to obtain media generation content, the media push content is obtained by merging the media generation content, and the media push content is output.

[0006] The task processing method provided in this embodiment firstly, based on the input media push material and prompt text, performs task arrangement on the material segments contained in the media push material to obtain the media generation task corresponding to the material segment. Then, based on the task parameters of the media generation task, it determines the appropriate resource configuration for the media generation task in the resource configuration of the heterogeneous resource pool. This achieves the determination of appropriate resource configuration for the media generation task corresponding to the material segment, improving the targeting and flexibility of subsequent task processing for the media generation task. Secondly, starting from the load data of heterogeneous resources under the appropriate resource configuration, it determines appropriate heterogeneous resources for the media generation task from the heterogeneous resources, thus enabling the media generation task to... Media generation is performed by assigning corresponding adapted heterogeneous resources to generate media content. This approach determines the appropriate adapted heterogeneous resources for different media generation tasks, thereby improving the targeting and accuracy of media generation task processing. Finally, the generated media content is merged to obtain media push content and output. This method utilizes the respective adapted heterogeneous resources to perform media generation tasks corresponding to the material segments, and performs distributed processing of media generation tasks to improve processing efficiency. By adapting heterogeneous resources to process the corresponding media generation tasks, the targeting of media generation tasks is improved, thereby enhancing the accuracy of the generated media push content.

[0007] Secondly, another embodiment of this specification provides a task processing apparatus, the apparatus comprising: The task orchestration module is used to orchestrate the material segments contained in the media push material according to the input media push material and prompt text, and obtain the media generation task corresponding to the material segment; The configuration determination module is used to determine the appropriate resource configuration in the resource configuration of the heterogeneous resource pool based on the task parameters of the media generation task. The resource determination module is used to determine the adaptive heterogeneous resources for the media generation task from the heterogeneous resources based on the load data of the heterogeneous resources under the adaptive resource configuration. The media generation module is used to assign the media generation task to the corresponding adapted heterogeneous resources to generate media content, merge the media generation content to obtain media push content, and output it.

[0008] Thirdly, in another embodiment of this specification, a task processing device is provided, the device comprising: a memory, a processor, and computer-executable instructions stored in the memory and executable on the processor, wherein the computer-executable instructions, when executed by the processor, implement the steps of the task processing method as described in the first aspect above.

[0009] Fourthly, in another embodiment of this specification, a computer-readable storage medium is provided for storing computer-executable instructions that, when executed by a processor, implement the steps of the task processing method described in the first aspect above.

[0010] Fifthly, in another embodiment of this specification, a computer program product is provided, the computer program product including a task processing program, the task processing program being executed by a processor to implement the steps of the task processing method as described in the first aspect above. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in one or more embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 A flowchart illustrating a task processing method provided in one embodiment of this specification; Figure 2 A schematic diagram of a task decision system provided in one embodiment of this specification; Figure 3 This is a schematic diagram illustrating a task processing procedure provided in one embodiment of this specification; Figure 4 A flowchart illustrating a task processing method for a media push scenario, provided as an embodiment of this specification; Figure 5 A schematic diagram of a task processing device provided in one embodiment of this specification; Figure 6 This is a schematic diagram of a task processing device provided in one embodiment of this specification. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0013] The following is an example of a task processing method provided in this specification: The task processing method provided in this embodiment can be executed by a computer device, which can be a terminal or a server. The terminal can include mobile phones, laptops, smart interactive devices, etc., and the server can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0014] Reference Figure 1 The task processing method provided in this embodiment specifically includes the following steps S102 to S108.

[0015] Step S102: Based on the input media push material and prompt text, perform task arrangement on the material segments contained in the media push material to obtain the media generation task corresponding to the material segment.

[0016] The media push materials mentioned in this embodiment refer to the materials required for media push. Media push materials can be user-inputted or uploaded, and can be used for media push in any media scenario, such as institutional media push materials, government push materials, educational push materials, news push materials, and / or advertising push materials. In addition, media push materials can also be media push materials for other media scenarios. The format of media push materials can be any format, such as text, audio, video, and / or images, such as institutional media push text, institutional media push audio, institutional media push video, and / or institutional media push images. Furthermore, media push materials may also include background materials, which can be materials used for background purposes, such as background images, background audio, or background video.

[0017] The prompt text can be a push notification based on media push materials. Specifically, it can be a prompt text indicating what kind of media push content should be generated based on the media push materials. For example, the prompt text could be "Please create a digital human explanation video based on the media push materials".

[0018] The media push material may contain one or more material segments; optionally, the material segments are obtained after slicing the media push material; specifically, the media push material can be divided into segments on an average basis to obtain material segments, or it can be divided into segments according to the index range and / or the material attributes of the media push material. The material attributes can be the functional attributes of the media push material, and media push materials with the same functional attributes can be grouped into the same material segment; for example, if the media push material is media push text, text segments can be obtained by segmenting the text according to the index segment number and / or the text attributes of the media push text. As another example, if the media push material is media push audio, audio segments can be obtained by segmenting the audio according to the pause points and / or the functional attributes of the media push audio. The pause points can be time points where the pause duration is greater than a duration threshold.

[0019] The media generation task corresponding to the material fragment can be any generation task involved in the media generation process based on the material fragment. The media generation task may include one or more sub-generation tasks, each representing a different task stage or node of the media generation task. For example, if the material fragment is an audio segment contained in the media push audio, and the prompt text is "Please create a digital human explanation video based on the media push material", then the media generation task corresponding to the material fragment includes an ASR (Automatic Speech Recognition) task (speech recognition task), a digital image generation task, a subtitle synthesis task, a sound cloning task, an audio synthesis task, and / or a video encoding task. The media generation tasks corresponding to each material fragment contained in the media push material may be the same or different.

[0020] In practice, the system can obtain the prompt text input by the user and the uploaded media push materials. The media push materials are then segmented to obtain material fragments. Based on the prompt text, the material fragments are arranged into tasks to obtain the media generation tasks corresponding to the material fragments. The media generation tasks corresponding to each material fragment can be the same or different. The user can be any user or a user corresponding to the media push category to which the media push material belongs. For example, if the media push category is an institutional media push category, the user can be an institutional member or a department member within the institution. If the media push category is a teaching push category, the user can be an institutional member of an educational institution. Specifically, in the process of arranging material fragments based on prompt text to obtain the corresponding media generation tasks, multiple functionally independent media generation tasks can be generated for the material fragments based on the prompt text; for example, image preprocessing tasks, feature extraction tasks, and classification tasks, which belong to different media generation tasks.

[0021] In the specific execution process, during the media generation task corresponding to the media clip, in order to ensure that the dependencies of each sub-generation task in the media generation task are satisfied and to maximize the parallel execution between sub-generation tasks, a task graph can be constructed. The task graph not only clearly displays each sub-generation task but also clearly shows the execution order of each sub-generation task, improving the efficiency and accuracy of task orchestration. In one optional implementation of this embodiment, during the process of orchestrating tasks for the media clips contained in the media push material based on the input media push material and prompt text to obtain the media generation task corresponding to the media clip, a task graph is constructed for the media clip based on the media push material and prompt text to obtain the task graph. Media generation tasks are then generated based on the task graph. Specifically, the following operations can be performed: Based on media push materials and prompt text, a task graph is constructed for the material fragments to obtain the task graph, and task keywords are extracted from the task graph. Based on the connection direction between sub-generated tasks corresponding to task keywords in the task graph, determine the task processing parameters of the sub-generated tasks. Media generation tasks are generated based on sub-generation tasks and task processing parameters.

[0022] The task keywords can be keywords representing sub-generated tasks; the task processing parameters of the sub-generated tasks can be execution parameters for executing the sub-generated tasks. These parameters may include the task processing order, task processing time, and / or a time threshold for task processing duration. Furthermore, the task processing parameters may include other types of processing parameters. The task graph can be a directed acyclic task graph, such as a task DAG (Directed Acyclic Graph).

[0023] Specifically, in determining the task processing parameters of sub-generated tasks based on the connection direction between sub-generated tasks corresponding to task keywords in the task graph, the task processing order of sub-generated tasks can be determined based on the connection direction between sub-generated tasks, and the task processing time and / or duration threshold of sub-generated tasks can be determined based on the task parameters of sub-generated tasks. The task parameters here can include task types, such as speech recognition, digital image generation, subtitle synthesis, speech cloning, audio synthesis and / or video encoding. The task parameters can also include the computing power and / or storage capacity required by the sub-generated tasks.

[0024] Step S104: Determine the appropriate resource configuration in the heterogeneous resource pool based on the task parameters of the media generation task.

[0025] Based on the input media push materials and prompt text, the above steps arrange the material segments contained in the media push materials into tasks to obtain the media generation tasks corresponding to the material segments. In this step, starting from the task parameters of the media generation task, the appropriate resource configuration is determined in the resource configuration of the heterogeneous resource pool. In this way, the task processing of the media generation task can be carried out more conveniently and quickly through the appropriate resource configuration.

[0026] The task parameters of the media generation task described in this embodiment may include the task type of the media generation task. The task type may be the task processing type that performs task processing on the media generation task. For example, the task type may include speech recognition, digital image generation, subtitle synthesis, speech cloning, audio synthesis, and / or video encoding. It should be noted that the examples of task types are merely illustrative. The actual task type of the media generation task is determined based on the specific task content of the media generation task or the task processing content of the media generation task. In addition, the task parameters of the media generation task may also include the computing power and / or storage capacity required by the media generation task.

[0027] The heterogeneous resource pool can be a pool composed of one or more heterogeneous resources. Heterogeneous resources can be resources that differ in processor architecture, computing power, storage technology, performance, and / or specifications. Heterogeneous resources can be hardware heterogeneous resources and / or software heterogeneous resources. Heterogeneous resources can be heterogeneous machine resources. For example, heterogeneous resources in the heterogeneous resource pool include ARM (Advanced RISC Machine) resources, GPU (Graphics Processing Unit) resources, ASIC (Graphics Processing Unit) resources, FPGA (Field-Programmable Gate Array) resources, and / or CPU (Central Processing Unit) resources. ARM resources can be machine resources that include ARM. GPU resources, ASIC resources, FPGA resources, and CPU resources are similar to ARM resources and will not be described in detail. In addition, heterogeneous resources can also include other types of hardware resources and / or software resources.

[0028] The resource configuration of the heterogeneous resource pool can be the resource configuration of the heterogeneous resources in the heterogeneous resource pool, that is, the resource configuration of all heterogeneous resources in the heterogeneous resource pool; the resource configuration can specifically be the machine model or machine type of the heterogeneous machine resources. For example, the heterogeneous resource pool includes ARM resources, GPU resources, CPU resources, FPGA resources and ASIC resources. The resource configuration of the heterogeneous resource pool includes ARM configuration, GPU configuration, CPU configuration, FPGA configuration and ASIC configuration. The resource configuration of the heterogeneous resource pool can be the sum of the resource configurations of the heterogeneous resources in the heterogeneous resource pool.

[0029] The adapted resource configuration can be a resource configuration selected from the resource configuration of the heterogeneous resource pool that is adapted to the media generation task. For example, the adapted resource configuration for the speech recognition task is CPU configuration, and the adapted resource configuration for the digital image generation task is GPU configuration.

[0030] In practice, there may be multiple media generation tasks. In order to improve the processing efficiency and speed of media generation tasks, and thus improve the processing efficiency of subsequent media generation and enhance the user experience, a distributed processing approach can be adopted. By distributing the sub-generation tasks of different stages of the media generation task to heterogeneous resources with different resource configurations, sufficient computing power for the media generation task can be ensured. At the same time, by determining the appropriate resource configuration for the media generation task, subsequent task processing of the media generation task can be carried out using heterogeneous resources under the appropriate resource configuration, which helps to improve the resource utilization of heterogeneous resources. By processing tasks using appropriate heterogeneous resources, the effectiveness of task processing of the media generation task can be improved. In one optional implementation of this embodiment, during the process of determining the appropriate resource configuration in the heterogeneous resource pool based on the task parameters of the media generation task, the appropriate resource configuration for the sub-generation tasks is determined in the resource configuration based on the computing power and storage capacity required by the sub-generation tasks in the media generation task. Specifically, the following operations can be performed: Determine the computing power and storage capacity required for sub-generation tasks within the media generation task; Based on computing power and storage capacity, the appropriate resource configuration for sub-generation tasks in the media generation task is determined in the resource configuration.

[0031] Among them, computing power can be the computing capacity required for the sub-generating task.

[0032] For example, the sub-generation tasks in the media generation task include speech recognition, digital image generation, subtitle synthesis, sound cloning, audio synthesis, and video encoding. The computing power and storage capacity required for each sub-generation task are determined. Based on the computing power and storage capacity, the appropriate resource configurations for each sub-generation task are determined in the resource configuration of the heterogeneous resource pool. The speech recognition task is configured with CPU, the digital image generation task, subtitle synthesis task, sound cloning task, and audio synthesis task are configured with GPU, and the video encoding task is configured with CPU.

[0033] For example, the video encoding task mentioned above is a video super-resolution task, which needs to super-resolution 1080P to 4K. The video super-resolution task includes video decoding, video super-resolution processing, video encoding, and merging tasks, which are distributed to different heterogeneous resources for processing. Based on the computing power and storage capacity required by the video super-resolution processing task, the adapted resource configuration for the video super-resolution processing task is determined to be GPU configuration, and the adapted resource configuration for the video decoding, video encoding, and merging tasks is determined to be CPU configuration.

[0034] Step S106: Based on the load data of heterogeneous resources under the adaptation resource configuration, determine the adaptation heterogeneous resources for the media generation task in the heterogeneous resources.

[0035] The above-mentioned task parameters based on the media generation task determine the appropriate resource configuration in the resource configuration of the heterogeneous resource pool. In this step, based on the load data of the heterogeneous resources under the appropriate resource configuration, the appropriate heterogeneous resources for the media generation task are determined in the heterogeneous resources under the appropriate resource configuration. In this way, the task processing of the media generation task is improved by adapting to heterogeneous resources, thereby improving the targeting and flexibility of task processing.

[0036] In this embodiment, the load data of heterogeneous resources can be the number of other media generation tasks undertaken by the heterogeneous resources, or it can be the resource utilization rate of the heterogeneous resources. For example, the load data of heterogeneous resources includes the memory occupancy and / or I / O requirements of the heterogeneous resources. The adapted heterogeneous resources can be heterogeneous resources adapted to the media generation tasks; the heterogeneous resources under the adapted resource configuration can belong to the heterogeneous resource pool.

[0037] In practice, when determining the appropriate heterogeneous resources for a media generation task based on the load data of heterogeneous resources under the appropriate resource configuration, if the appropriate resource configurations of the sub-generation tasks in the media generation task are the same, the task priority of the sub-generation task can be determined based on the task parameters, urgency and / or user requirements of the sub-generation task, the resource priority of the heterogeneous resource can be determined based on the load data of the heterogeneous resource, and the sub-generation task and the heterogeneous resource can be matched based on the task priority and the selection priority to obtain the appropriate heterogeneous resources for the sub-generation task.

[0038] Step S108: Assign the media generation task to the corresponding adapted heterogeneous resources to generate media content, merge the media generation content to obtain media push content and output it.

[0039] Based on the load data of heterogeneous resources under the adaptation resource configuration, the adaptation heterogeneous resources for media generation tasks are determined in the heterogeneous resources. In this step, the media generation tasks are assigned to the corresponding adaptation heterogeneous resources, and the media generation content is obtained by generating media based on the media generation tasks through the adaptation heterogeneous resources. The media generation content is then merged to obtain the media push content and output.

[0040] The media-generated content described in this embodiment can be the result of task processing obtained by performing task processing on a media generation task. For example, the media-generated content obtained by performing media generation on a media generation task is a digital human narration video generated based on each audio segment of the media push audio.

[0041] The media push content refers to the merged result obtained by merging media-generated content; using the previous example, the media push content is the merged digital human narration video obtained by merging the digital human narration videos generated based on each audio segment.

[0042] In practice, during the process of assigning media generation tasks to corresponding adapted heterogeneous resources to generate media content, the media generation tasks can be assigned to corresponding adapted heterogeneous resources according to the task processing parameters of the sub-generation tasks in the media generation task corresponding to each material segment, thereby obtaining the media generation content of each material segment; during the process of merging the media generation content to obtain media push content, the media generation content of each material segment can be merged to obtain media push content.

[0043] In the specific execution process, in order to improve the task processing efficiency of media generation tasks, a distributed processing approach is adopted to process media generation tasks. This is achieved by adapting media generation tasks to corresponding heterogeneous resources, thereby improving the processing efficiency and accuracy of media generation tasks. In one optional implementation of this embodiment, during the process of allocating media generation tasks to corresponding adapted heterogeneous resources to generate media content, the following operations are performed: The speech recognition task in the media generation task is allocated to CPU resources to perform speech recognition of audio segments of media push audio to obtain text segments. The digital image generation task in the media generation task is allocated to GPU resources to generate digital images, thereby obtaining the digital images under the media push category to which the media push audio belongs. Generate media broadcast videos based on text snippets, digital images, and audio snippets.

[0044] Based on this, in order to further improve the efficiency and accuracy of generating media broadcast videos, in an optional implementation of this embodiment, the following operations are performed during the process of generating media broadcast videos based on text fragments, digital images, and audio fragments: The digital avatar video is obtained by compositing digital avatars and background videos using GPU resources, and the composite video is obtained by compositing subtitles based on text fragments. The system uses GPU resources to clone audio segments, generates broadcast audio based on the cloned audio parameters and text segments, and synthesizes the broadcast audio in the video to obtain the broadcast video. The media broadcast video is obtained by encoding the broadcast video using CPU resources.

[0045] Voice cloning can be based on audio segments for voice cloning or voice parameter cloning. Voice parameters can include audio frequency, pitch, speech rate, and / or amplitude. Other types of parameters may also be included. Video encoding can include narrowband high-definition encoding or video quality enhancement.

[0046] Specifically, in the process of video encoding the broadcast video to obtain the media broadcast video, multiple sets of video frames in the broadcast video can be processed in parallel, that is, multiple sets of video frames can be processed in parallel at the same time; for example, multiple sets of video frames are multiple GOPs (Group of Pictures, image groups in video compression formats), multiple GOPs in the broadcast video can be processed in parallel, and the parallel processing results can be merged to obtain the media broadcast video; In the process of encoding the broadcast video to obtain the media broadcast video, each video frame in the broadcast video can also be processed in parallel, and the results of the parallel processing can be merged to obtain the media broadcast video.

[0047] After obtaining the media-generated content, i.e., the media-generated content of each material segment, the media-generated content of each material segment can be merged to obtain the media push content. To ensure the accuracy and completeness of the merged media push content, multiple verification methods can be used. If problems exist, some sub-generation tasks in the media generation task can be re-executed or the algorithm parameters used in the media generation task can be adjusted.

[0048] In practical applications, sub-generation tasks within a media generation task may encounter processing anomalies. Failure to respond promptly to these anomalies can lead to low resource utilization. To address this, in one optional implementation of this embodiment, the following operations are also performed: If any task processing abnormality is detected in any sub-generated task of the media generation task, or if the resource requirement of any sub-generated task exceeds the requirement threshold, the media generation task will be paused or heterogeneous resource matching will be performed again for any sub-generated task. or, If the processing time of any sub-generation task exceeds the time threshold, the processing of the media generation task will be terminated.

[0049] The duration threshold can be the duration threshold of the task processing duration of the sub-generation task included in the task processing parameters of the sub-generation task in the media generation task; heterogeneous resource matching can be to re-determine the adapted heterogeneous resources of any sub-generation task in the heterogeneous resources based on the load data of the heterogeneous resources under the adapted resource configuration of any sub-generation task, and re-process any sub-generation task by adapting the heterogeneous resources.

[0050] This embodiment can be applied to task decision-making systems or media engines, specifically intelligent task decision-making systems or intelligent media engines, such as AIGC (Artificial Intelligence Generated Content) systems or Worker-Brain (intelligent units responsible for executing specific tasks) media engines; for example... Figure 2 As shown, a distributed architecture is adopted for distributed processing to improve multi-machine parallelism. During task orchestration, a directed acyclic graph (DAG) is used for task orchestration. In the process of processing the video encoding tasks included in the media generation task, GOP-level parallel processing and YUV (color encoding model) frame-level parallel processing can be used to obtain the media playback video. In the parallel or distributed processing of the media generation task, a unified media processing framework and a unified algorithm underlying dependency library are used. That is, when each heterogeneous resource is processed, the media processing framework and algorithm underlying dependency library used are the same. In terms of algorithms, hardware and software integration optimization can be achieved to improve algorithm performance. Through CPU, GPU and ASCII encoding and decoding acceleration, it supports multiple machine models and specifications at different times in the cloud. At the same time, the single frame processing performance is optimized from both algorithm and engineering dimensions.

[0051] For example Figure 3As shown, the media push materials include spoken audio and background video. The prompt text is "Please create a digital human explanation video based on the spoken audio and background video." Based on the prompt text and spoken audio, tasks are arranged for the audio segments contained in the spoken audio. The corresponding media generation tasks for each audio segment include ASR tasks, digital human generation tasks, subtitle synthesis tasks, sound cloning tasks, audio synthesis tasks, and video encoding tasks. Based on the task type of the media generation task for each audio segment, an appropriate resource configuration is determined in the heterogeneous resource pool. Based on the load data of the heterogeneous resources under the appropriate resource configuration, appropriate heterogeneous resources for the media generation tasks are determined. The spoken audio includes audio segment 1, audio segment 2, up to audio segment m. The media generation task for each audio segment is assigned to the corresponding appropriate heterogeneous resources for media generation to obtain media generated content. The media generated content is merged to obtain the media push content and output. Distributed media is used. The media processing engine performs media generation tasks for each audio segment to obtain media generated content. It then merges the media generated content to obtain media push content and outputs it. Specifically, it allocates the ASR task for each audio segment to CPU resources for speech recognition to obtain text segments, and allocates the digital human task to GPU resources for digital human image generation to obtain the digital human image under the media push category to which the spoken audio belongs. It then uses GPU resources to synthesize the digital human image and background video to obtain a synthesized video. Based on the text segments, it completes the subtitle synthesis task in the synthesized video, completes the sound cloning task based on the audio segments, and generates broadcast audio based on the cloned sound parameters and text segments. Finally, it synthesizes the broadcast audio in the synthesized video to obtain a broadcast video, and uses CPU resources to complete the narrowband high-definition encoding task for the broadcast video to obtain the digital human video corresponding to each audio segment. Finally, it merges the digital human videos to obtain a digital human broadcast video or a digital human explanation video.

[0052] The task decision-making system or media engine in this embodiment may include a production module, a media asset module, and a media processing module. The production module can be used to shift from using AI (Artificial Intelligence) technology in a single production stage to fully intelligent production. The process of task arrangement and media generation to obtain media content can all adopt a large model. The media asset module can more intelligently understand media asset content and use natural language for search and management. The media processing module can use models to achieve maximum effect optimization, including high-definition enhanced scenes, low-definition rebirth scenes, and scenes that use dual-channel audio devices to obtain panoramic or surround sound effects.

[0053] The task decision system can also use the provided interfaces or tools to detect the CPU usage of heterogeneous resources in real time or periodically. CPU usage can include overall utilization, utilization of each core, and / or load balancing. It can also detect the resource requirements of running media generation tasks periodically or in real time. Resource requirements can include CPU utilization, memory usage, and / or I / O requirements. CPU usage and resource requirements of media generation tasks can be collected through process management tools or specific performance monitoring software. In the process of determining the appropriate heterogeneous resources for media generation tasks based on the load data of heterogeneous resources under the adaptation resource configuration, the importance of media generation tasks can be determined. For example, based on the resource requirements of media generation tasks and the CPU usage of heterogeneous resources, the availability of heterogeneous resources can be evaluated to determine whether there are enough idle heterogeneous resources to process new media generation tasks, or whether it is necessary to adjust the running media generation tasks to release heterogeneous resources. The priority of media generation tasks is determined based on their importance, nature, urgency, and / or resource requirements. Based on the priority of the media generation tasks and the load data of heterogeneous resources under the adapted resource configuration, the appropriate heterogeneous resources for each media generation task are determined. For example, a priority is assigned to each media generation task, with higher-priority tasks receiving priority access to heterogeneous resources. When multiple media generation tasks compete for heterogeneous resources simultaneously, the needs of higher-priority tasks are prioritized. Media generation tasks with high real-time requirements and critical business tasks can be set to high priority.

[0054] In determining the appropriate heterogeneous resources for media generation tasks based on the load data of heterogeneous resources under the adapted resource configuration, the current load of each core of the heterogeneous resources can also be considered. Media generation tasks can be allocated to cores with lighter loads. Load balancing algorithms, such as round-robin and least-connection methods, can be used to determine the target cores for media generation task allocation. The allocation of media generation tasks on heterogeneous resources can be dynamically adjusted according to the execution status of media generation tasks, such as changes in execution progress and resource requirements. Specifically, if the resource requirements of the target media generation task increase, more CPU time can be allocated to the target media generation task; if the resource requirements of the target media generation task decrease, the remaining heterogeneous resources are allocated to other media generation tasks. During the process of allocating media generation tasks to corresponding adapted heterogeneous resources to generate media content, a task scheduling algorithm can be used to schedule the media generation tasks to the corresponding adapted heterogeneous resources to generate media content. The task scheduling algorithm can include First-Come, First-Served (FCFS), Shortest Job First (SJF), or Round-Robin, determined based on the task characteristics and resource requirements of the media generation task. In this process, algorithm or model performance can also be optimized. Based on the execution status of the media generation task and user feedback, the relevant algorithms and models can be adjusted and improved. Performance data and user feedback from the task decision system are regularly summarized, and algorithms and models are optimized based on the performance data and user feedback.

[0055] Furthermore, a management mechanism for media generation tasks can be established. This mechanism may include task status tracking, progress detection, and / or error handling. Data structures such as task queues and status tables can be used to record information about media generation tasks. This embodiment can also optimize the task orchestration strategy, reduce the dependencies between sub-generation tasks in media generation tasks, and improve parallelism. The scheduling algorithm for sub-generation tasks can be improved to better utilize computing resources and reduce task execution time. The algorithm should be designed with scalability and adaptability in mind to handle tasks of different scales and types. The algorithm can be easily adjusted and extended through parametric design, modular architecture, and other methods.

[0056] It should be added that each optional implementation method or each feasible execution method in steps S102 to S108 provided in this embodiment can be executed independently as needed, or they can be combined or referenced with each other. At the same time, each specific execution step in each optional implementation method or each feasible execution method can also be executed independently, and this embodiment does not make specific limitations.

[0057] In summary, the task processing method provided in this embodiment constructs a task graph based on the input media push material and prompt text, extracting task keywords from the task graph. Based on the connection direction between sub-generation tasks corresponding to the task keywords in the task graph, it determines the task processing parameters of the sub-generation tasks. Based on the sub-generation tasks and task processing parameters, it generates media generation tasks corresponding to the media push material segments. According to the task parameters of the media generation tasks, it determines the appropriate resource configuration for the media generation tasks in the heterogeneous resource pool, thereby improving the targeting and flexibility of subsequent task processing for media generation tasks. Furthermore, it adapts the resource configuration to the media generation tasks. Starting with the load data of heterogeneous resources under resource configuration, suitable heterogeneous resources are determined for media generation tasks. These tasks are then assigned to the corresponding suitable heterogeneous resources to generate media content. This approach determines suitable heterogeneous resources for different media generation tasks, thereby improving the targeting and accuracy of media generation task processing. Finally, the generated media content is merged to obtain media push content and output. This method utilizes suitable heterogeneous resources to generate media for material segments, performing distributed processing to improve the efficiency of media generation tasks. By using suitable heterogeneous resources to process corresponding media generation tasks, the targeting of media generation tasks is improved, ultimately enhancing the accuracy of the generated media push content.

[0058] The following combination Figure 4 Taking the application of the task processing method provided in this embodiment in a media push scenario as an example, the task processing method provided in this embodiment will be further explained. (Refer to...) Figure 4 The task processing method applied to media push scenarios includes the following steps.

[0059] Step S402: Based on the input media push audio and prompt text, construct a task graph for the audio segments contained in the media push audio to obtain a task graph.

[0060] Step S404: Extract task keywords from the task graph and determine the task processing parameters of the sub-generated tasks based on the connection direction between the sub-generated tasks corresponding to the task keywords in the task graph.

[0061] Step S406: Based on the sub-generation task and task processing parameters, generate a media generation task corresponding to the audio segments contained in the media push audio.

[0062] Step S408: Determine the computing power and storage capacity required for the sub-generation tasks in the media generation task.

[0063] Step S410: Based on computing power and storage capacity, determine the appropriate resource configuration for sub-generation tasks in the media generation task within the resource configuration of the heterogeneous resource pool.

[0064] Step S412: Based on the load data of heterogeneous resources under the adaptation resource configuration, determine the adaptation heterogeneous resources for the media generation task corresponding to each audio segment in the heterogeneous resources.

[0065] Step S414: Assign the media generation task corresponding to each audio segment to the corresponding adapted heterogeneous resources to generate media content, merge the media generation content to obtain media push content and output it.

[0066] It should be noted that any one or any combination of steps S402 to S414 can be replaced by the corresponding technical means provided in steps S102 to S108 as needed for implementation and deployment, which will not be elaborated here.

[0067] Figure 5 This is a schematic diagram of a task processing device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the device includes: The task orchestration module 502 is used to orchestrate the material segments contained in the media push material according to the input media push material and prompt text, and obtain the media generation task corresponding to the material segment; The configuration determination module 504 is used to determine the appropriate resource configuration in the resource configuration of the heterogeneous resource pool based on the task parameters of the media generation task. The resource determination module 506 is used to determine the adaptive heterogeneous resources for the media generation task from the heterogeneous resources based on the load data of the heterogeneous resources under the adaptive resource configuration. The media generation module 508 is used to assign the media generation task to the corresponding adapted heterogeneous resources to generate media content, merge the media generation content to obtain media push content and output it.

[0068] The task processing device provided in this embodiment first runs the task orchestration module 502 to orchestrate the material segments contained in the media push material according to the input media push material and prompt text, thereby obtaining the media generation task corresponding to the material segment. Then, it runs the configuration determination module 504 to determine the appropriate resource configuration in the resource configuration of the heterogeneous resource pool based on the task parameters of the media generation task. Next, it runs the resource determination module 506 to determine the appropriate heterogeneous resource for the media generation task in the heterogeneous resources according to the load data of the heterogeneous resources under the appropriate resource configuration. Finally, it runs the media generation module 508 to allocate the media generation task to the corresponding appropriate heterogeneous resource for media generation to obtain media generation content. The media generation content is then merged to obtain media push content and output, thereby improving the accuracy of the media push content.

[0069] The task processing apparatus provided in one embodiment of this specification can implement the various processes in the foregoing method embodiments and achieve the same functions and effects, which will not be repeated here.

[0070] Furthermore, one embodiment of this specification also provides a task processing device. Figure 6 This is a schematic diagram of the structure of a task processing device provided in one embodiment of this specification, as shown below. Figure 6 As shown, the device includes a memory 601, a processor 602, a bus 603, and a communication interface 604. The memory 601, the processor 602, and the communication interface 604 communicate via the bus 603. The communication interface 604 may include input / output interfaces, including but not limited to a keyboard, mouse, monitor, microphone, and loudspeaker.

[0071] Figure 6 In the processor 602, the memory 601 stores computer-executable instructions that can run on the processor 602. When the processor 602 executes the computer-executable instructions, the following process is implemented: Based on the input media push materials and prompt text, the material fragments contained in the media push materials are arranged into tasks to obtain the media generation tasks corresponding to the material fragments; Based on the task parameters of the media generation task, the appropriate resource configuration is determined in the resource configuration of the heterogeneous resource pool; Based on the load data of the heterogeneous resources under the adapted resource configuration, the adapted heterogeneous resources for the media generation task are determined from the heterogeneous resources. The media generation task is assigned to the corresponding adapted heterogeneous resources to generate media content. The generated media content is then merged to obtain media push content and output.

[0072] The task processing device provided in this embodiment, through the cooperation of memory 601, processor 602, bus 603 and communication interface 604, arranges the material segments contained in the media push material according to the input media push material and prompt text, and obtains the media generation task corresponding to the material segment; determines the appropriate resource configuration in the resource configuration of the heterogeneous resource pool based on the task parameters of the media generation task; determines the appropriate heterogeneous resource for the media generation task in the heterogeneous resources according to the load data of the heterogeneous resources under the appropriate resource configuration; assigns the media generation task to the corresponding appropriate heterogeneous resource to generate media content, merges the media generation content to obtain media push content and outputs it, so as to improve the accuracy of the media push content.

[0073] The task processing device provided in one embodiment of this specification can implement the various processes in the foregoing method embodiments and achieve the same functions and effects, which will not be repeated here.

[0074] Furthermore, another embodiment of this specification provides a computer-readable storage medium for storing computer-executable instructions that, when executed by a processor, implement the following process: Based on the input media push materials and prompt text, the material fragments contained in the media push materials are arranged into tasks to obtain the media generation tasks corresponding to the material fragments; Based on the task parameters of the media generation task, the appropriate resource configuration is determined in the resource configuration of the heterogeneous resource pool; Based on the load data of the heterogeneous resources under the adapted resource configuration, the adapted heterogeneous resources for the media generation task are determined from the heterogeneous resources. The media generation task is assigned to the corresponding adapted heterogeneous resources to generate media content. The generated media content is then merged to obtain media push content and output.

[0075] The computer-readable storage medium provided in this embodiment performs task arrangement on the media push materials and prompt text, and obtains the media generation task corresponding to the media push materials. Based on the task parameters of the media generation task, it determines the appropriate resource configuration in the resource configuration of the heterogeneous resource pool. Based on the load data of the heterogeneous resources under the appropriate resource configuration, it determines the appropriate heterogeneous resources for the media generation task in the heterogeneous resources. It assigns the media generation task to the corresponding appropriate heterogeneous resources to generate media content, merges the media generation content to obtain media push content, and outputs it to improve the accuracy of the media push content.

[0076] The computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc.

[0077] An embodiment of this specification provides a computer-readable storage medium that can implement the various processes in the foregoing method embodiments and achieve the same functions and effects, which will not be repeated here.

[0078] Furthermore, another embodiment of this specification provides a computer program product that, when executed by a processor, implements the following process: Based on the input media push materials and prompt text, the material fragments contained in the media push materials are arranged into tasks to obtain the media generation tasks corresponding to the material fragments; Based on the task parameters of the media generation task, the appropriate resource configuration is determined in the resource configuration of the heterogeneous resource pool; Based on the load data of the heterogeneous resources under the adapted resource configuration, the adapted heterogeneous resources for the media generation task are determined from the heterogeneous resources. The media generation task is assigned to the corresponding adapted heterogeneous resources to generate media content. The generated media content is then merged to obtain media push content and output.

[0079] The computer program product provided in this embodiment performs task arrangement on the media push materials and prompt text, which contain material fragments, to obtain media generation tasks corresponding to the material fragments. Based on the task parameters of the media generation tasks, it determines the appropriate resource configuration in the resource configuration of the heterogeneous resource pool. Based on the load data of the heterogeneous resources under the appropriate resource configuration, it determines the appropriate heterogeneous resources for the media generation tasks in the heterogeneous resources. It assigns the media generation tasks to the corresponding appropriate heterogeneous resources to generate media content, merges the media generation content to obtain media push content, and outputs it to improve the accuracy of the media push content.

[0080] An embodiment of this specification provides a computer program product that can implement the various processes in the foregoing method embodiments and achieve the same functions and effects, which will not be repeated here.

[0081] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0086] Memory may include non-persistent storage in computer-readable storage media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable storage media.

[0087] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0088] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A task processing method, characterized in that, The method includes: Based on the input media push materials and prompt text, the material fragments contained in the media push materials are arranged into tasks to obtain the media generation tasks corresponding to the material fragments; Based on the task parameters of the media generation task, the appropriate resource configuration is determined in the resource configuration of the heterogeneous resource pool; Based on the load data of the heterogeneous resources under the adapted resource configuration, the adapted heterogeneous resources for the media generation task are determined from the heterogeneous resources. The media generation task is assigned to the corresponding adapted heterogeneous resources to generate media content. The generated media content is then merged to obtain media push content and output.

2. The task processing method according to claim 1, characterized in that, The step of arranging tasks for media clips contained in the media push materials based on the input media push materials and prompt text, and obtaining the media generation tasks corresponding to the media clips, includes: Based on the media push materials and the prompt text, a task graph is constructed for the material fragments to obtain the task graph, and task keywords are extracted from the task graph. Based on the connection direction between the sub-generated tasks corresponding to the task keywords in the task graph, the task processing parameters of the sub-generated tasks are determined. The media generation task is generated based on the sub-generation task and the task processing parameters.

3. The task processing method according to claim 2, characterized in that, The process of determining the appropriate resource configuration in the heterogeneous resource pool based on the task parameters of the media generation task includes: Determine the computing power and storage capacity required for the sub-generation tasks in the media generation task; Based on the computing power and the storage capacity, the appropriate resource configuration for the sub-generation tasks in the media generation task is determined in the resource configuration.

4. The task processing method according to claim 1, characterized in that, The step of assigning the media generation task to the corresponding adapted heterogeneous resources to generate media content includes: The speech recognition task in the media generation task is allocated to CPU resources to perform speech recognition of audio segments of media push audio to obtain text segments; The digital image generation task in the media generation task is allocated to GPU resources to generate the digital image, thereby obtaining the digital image under the media push category to which the media push audio belongs; A media broadcast video is generated based on the text fragment, the digital image, and the audio fragment.

5. The task processing method according to claim 4, characterized in that, The step of generating a media broadcast video based on the text fragment, the digital image, and the audio fragment includes: The digital image video is obtained by compositing the digital image and background video using GPU resources, and the digital image video is then used to synthesize subtitles based on the text fragments to obtain a composite video. The audio segment is cloned using GPU resources, and a broadcast audio is generated based on the cloned audio parameters and the text segment. The broadcast audio is then synthesized in the synthesized video to obtain the broadcast video. The media broadcast video is obtained by video encoding the broadcast video using CPU resources.

6. The task processing method according to claim 1, characterized in that, The method further includes: If any task processing abnormality is detected in any sub-generated task of the media generation task or the resource requirement of any sub-generated task exceeds the requirement threshold, the media generation task is paused or heterogeneous resource matching is re-performed for any sub-generated task. or, If the processing time of any of the sub-generation tasks exceeds the time threshold, the processing of the media generation task is terminated.

7. A task processing device, characterized in that, The device includes: The task orchestration module is used to orchestrate the material segments contained in the media push material according to the input media push material and prompt text, and obtain the media generation task corresponding to the material segment; The configuration determination module is used to determine the appropriate resource configuration in the resource configuration of the heterogeneous resource pool based on the task parameters of the media generation task. The resource determination module is used to determine the adaptive heterogeneous resources for the media generation task from the heterogeneous resources based on the load data of the heterogeneous resources under the adaptive resource configuration. The media generation module is used to assign the media generation task to the corresponding adapted heterogeneous resources to generate media content, merge the media generation content to obtain media push content, and output it.

8. A task processing device, characterized in that, The device includes a memory and a processor, the memory storing computer-executable instructions that, when executed on the processor, enable the implementation of the steps of the method described in any one of claims 1-6.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by a processor, they can implement the steps of the method described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a task processing program, which, when executed by a processor, is capable of implementing the steps of the method described in any one of claims 1-6.

Images