Model reasoning method and system based on high-concurrency plug flow and storage medium
By introducing dynamic resource management and optimization processing mechanisms into high-concurrent thrust flow systems, the problems of low resource utilization and low processing efficiency in the existing technology are solved, and efficient and reliable high-concurrent thrust flow processing is achieved.
Patent Information
- Application Number
- CN202510148232.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-11
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art has problems such as low resource utilization, low processing efficiency, poor reliability and poor scalability when dealing with high concurrent push flows, which is difficult to meet the needs of practical applications.
A model inference method based on high concurrent push flow is adopted, and by receiving push flow data from multiple clients, preprocessing and resource management, dynamically allocating and managing system resources, realizing resource pooling and priority queue mechanisms, improving resource utilization and processing efficiency.
It realizes efficient processing of high concurrent requests, improves resource management efficiency and system reliability, and enhances scalability and user experience.
Smart Images
Figure CN119893175A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of streaming data processing, and in particular to a model reasoning method, system and storage medium based on high-concurrency streaming. Background Art
[0002] With the rapid development of the Internet and multimedia technologies, application scenarios such as live video and real-time communication are becoming more and more common. In particular, in scenarios such as large-scale live events, online education, and remote conferences, the demand for high-concurrency streaming is becoming more and more significant. However, existing technologies face many challenges in handling high-concurrency streaming and are difficult to meet the needs of actual applications.
[0003] The current high-concurrency streaming method mainly relies on increasing the number of servers and improving hardware performance to cope with a large number of concurrent requests. Although this method has alleviated the pressure brought by high concurrency to a certain extent, its problems such as low resource utilization, low processing efficiency, poor reliability and stability, and poor scalability still exist. Traditional methods increase processing power by increasing hardware resources, but often fail to make full use of existing hardware resources, resulting in resource waste. In the case of high concurrency, resource allocation is uneven, some servers may be overloaded, while other server resources are idle, and dynamic scheduling and optimization of resources cannot be achieved.
[0004] Existing methods usually use single-thread or simple multi-thread methods to process streaming requests, which makes it difficult to effectively utilize the parallel computing capabilities of modern multi-core processors. The data reception and processing process is complex and time-consuming, especially when facing a large number of video streams. The processing efficiency is low, which easily leads to data backlog and delay. In high concurrency situations, the system is prone to bottlenecks and single points of failure, affecting the overall service quality and user experience. The lack of effective load balancing and fault recovery mechanisms leads to unstable system performance and even crashes in some cases.
[0005] In recent years, the introduction of load balancing and fault recovery mechanisms can ensure the stability and reliability of the system under high concurrency. Through redundant design and real-time monitoring, system failures can be discovered and handled in a timely manner to ensure service continuity and stability. Containerization and microservice architecture can be used to achieve flexible expansion and dynamic adjustment of the system and reduce expansion costs. According to actual concurrency requirements, system resources can be dynamically increased or decreased to flexibly respond to concurrent requests of different scales.
[0006] In summary, although the existing technology has solved the problem of high-concurrency streaming to a certain extent, it still has shortcomings such as low resource utilization, low processing efficiency, poor reliability and poor scalability. Summary of the invention
[0007] In view of the deficiencies in the prior art, the object of the present invention is to provide a model reasoning method, system and storage medium based on high-concurrency streaming.
[0008] In order to achieve the above object, the present invention provides the following technical solutions:
[0009] A model reasoning method based on high-concurrency streaming includes the following steps:
[0010] Receive streaming data from several clients;
[0011] Preprocess the received streaming data to extract latent variables of character video frames, audio data, facial data, and facial images;
[0012] Extract audio feature data from the preprocessed audio data, synthesize the audio feature data and the latent variables of the face image, generate an image that is consistent with the lip shape and audio data, and fuse the image with the original video frame of the streaming data to obtain the final video frame;
[0013] Convert the final video frame into audio and video streams and push them to the designated streaming server via RTMP protocol, using load balancing and caching technology to perform multiple streaming tasks simultaneously;
[0014] In the step of obtaining the final video frame, system resources are dynamically allocated and managed, a list of available CUDA devices and model resources is maintained, and a resource pool and priority queue mechanism are implemented.
[0015] In the present invention, preferably, the streaming data from multiple clients is received through the Flask framework, and the Flask framework adopts the HTTP POST interface to receive audio, video and text data. A data parsing and verification unit is provided in the Flask framework to perform preliminary parsing and verification on the received request data to ensure the integrity and correctness of the data.
[0016] In the present invention, preferably, the received streaming data is preprocessed by a preprocessing module, and the preprocessing module includes a video frame extraction unit, a text-to-speech unit, a facial data loading unit and a latent variable extraction unit. The video frame extraction unit uses FFmpeg to extract the character image video into a video frame image, the text-to-speech unit converts the text data into audio data, the facial data loading unit is used to perform face detection, and extract facial key points, lip area masks and facial latent variables to represent facial related data, and the latent variable extraction unit is used to extract latent variables of the face image.
[0017] In the present invention, preferably, a resource allocation module is used to dynamically allocate and manage system resources. The resource allocation module includes a resource management unit. The resource management unit monitors the usage of each CUDA device in real time, dynamically adjusts the allocation strategy of tasks on the CUDA device, and allocates model resources according to the size of the model and the priority of the task.
[0018] In the present invention, preferably, the resource allocation module further includes a task ID allocation unit, which allocates a unique task ID to each streaming task for identifying the task and tracking its execution status and resource usage.
[0019] In the present invention, preferably, the resource allocation module further includes a resource recovery unit, and the resource recovery unit re-adds the released resources to the available resource list after the current streaming task is completed.
[0020] In the present invention, preferably, the preprocessed audio data is processed by an inference module to obtain a final video frame, and the inference module includes an audio processing unit, a data alignment unit, and a data generator. The audio processing unit is used to extract audio feature data from the audio data, the data alignment unit aligns the audio feature data and the video frame data, and the data generator obtains the aligned audio features and latent variables.
[0021] In the present invention, preferably, the inference module also includes a variational autoencoder unit, a UNET unit and a synthesis unit. The UNET unit receives the audio features and latent variables output by the data generator to generate latent variables of a face image with consistent lip shape and audio. The variational autoencoder unit decodes the latent variables of the face image and reconstructs an image with the same lip shape and audio data as the original image data size. The synthesis unit combines the image with the original video frame to generate a final video frame with a lip synchronization effect.
[0022] A model reasoning system based on high-concurrency push streaming, including a data receiving module, a preprocessing module, a resource allocation module, a reasoning module and a result output module;
[0023] The preprocessing module includes a video frame extraction unit, a text-to-speech unit, a facial data loading unit, and a latent variable extraction unit;
[0024] The resource allocation module includes a resource management unit, a task ID allocation unit and a resource recovery unit;
[0025] The inference module includes an audio processing unit, a data alignment unit, a data generator, a variational autoencoder unit, a UNET unit and a synthesis unit;
[0026] The result output module is composed of an audio and video synthesis unit, an RTMP streaming unit, a load balancing unit and a cache unit.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] (1) Efficient processing of high-concurrency requests: By optimizing the data receiving and processing mechanism, it can efficiently handle high-concurrency streaming requests from multiple clients.
[0029] (2) Resource management optimization: By dynamically allocating and managing system resources, we can improve resource utilization and ensure timely processing of high-priority tasks.
[0030] (3) Improved reasoning efficiency: Parallel computing and asynchronous processing technologies are used to significantly improve the processing speed and efficiency of the reasoning module.
[0031] (4) Efficient output results: Through load balancing and caching technology, the efficient output of the inference module is ensured, improving the overall performance of the system.
[0032] (5) Enhanced reliability: By introducing load balancing and fault recovery mechanisms, the stability and reliability of the system of the present invention are enhanced under high concurrency conditions.
[0033] (6) Improved scalability: Utilizing containerization and microservice architecture, flexible expansion and dynamic adjustment are achieved, expansion costs are reduced, and the system can flexibly respond to concurrent requests of different scales. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 The present invention is a flowchart of a model reasoning method based on high-concurrency streaming.
[0035] Figure 2 This is a schematic diagram of the detailed processing process of a model reasoning method based on high-concurrency streaming described in the present invention.
[0036] Figure 3 The resource allocation process diagram of the resource management module of the present invention is shown in FIG.
[0037] Figure 4 It is a schematic diagram of the reasoning flow of the reasoning module described in the present invention.
[0038] Figure 5 It is a flow chart of the result output module of the present invention.
[0039] Figure 6 It is a schematic diagram of key points of a face of the present invention. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.
[0042] Please also see Figures 1 to 5 , a preferred embodiment of the present invention provides a model reasoning method based on high-concurrency push streaming, in view of the fact that application scenarios such as video live broadcast and real-time communication are becoming more and more common in current Internet applications, the demand for high-concurrency push streaming is becoming more and more significant. For example, in the live broadcast of major events, popular game live broadcasts or popular online classes, a large number of viewers often pour in at the same time, which places extremely high demands on the processing power of the server. When processing high-concurrency push streaming requests, the prior art usually relies on increasing the number of servers and improving hardware performance to improve processing power. However, this method has significant deficiencies in resource utilization, processing efficiency, system reliability and scalability. A new high-concurrency push streaming model reasoning method proposed by the present invention realizes efficient processing of high-concurrency push streaming requests by introducing parallel computing technology and optimized resource management mechanism. Specifically, the innovation of the present invention lies in the dynamic resource allocation mechanism, which intelligently allocates resources according to task priority and system load to ensure efficient and stable streaming media push, aiming to solve the problems of low resource utilization, low processing efficiency, poor reliability and poor scalability in the prior art when processing high-concurrency push streaming requests. By introducing parallel computing and optimizing resource management technology, the present invention can significantly improve streaming processing capabilities, resource management efficiency and system reliability.
[0043] The specific steps include:
[0044] S0. Task image video data preparation: Use any camera device to shoot a full-body or half-body video of any length. The video format requires 25 frames. The face must be clearly recognizable and the size of the face must account for more than 10% of the video frame. The front face must always face the recording device. The head cannot twist or shake too much. Only one image is allowed in the image video.
[0045] Text and voice data preparation: Convert the information that you want the character to express during the live broadcast into text or voice data. If the text or voice data is too long, it needs to be segmented. The text is divided into segments of about 500 words, and the voice data is divided into segments of about 120 seconds. The text can be directly generated by a large language model such as Llama3. The voice data can be recorded using any recording device in .wav format.
[0046] In the live video broadcast scenario, the user broadcasts the live video through the platform client and sends the prepared character image video, audio or text data to the server.
[0047] S1. After the data receiving module receives the data sending request from the client, the server downloads the audio, video and text data and saves them to a local temporary path. The Flask framework is used to implement the API interface, supporting multi-threaded or multi-process mode to handle data upload requests from multiple clients, ensuring that the system can cope with the pressure brought by high concurrent requests. The Flask framework receives audio, video and text data through the HTTP POST interface. The Flask framework is also equipped with a data parsing and verification unit to perform preliminary parsing and verification on the received request data to ensure the integrity and correctness of the data.
[0048] The received data is input into the preprocessing module for preprocessing, which includes audio feature extraction, text-to-speech, and facial data extraction. Task queues and parallel processing technology are used to ensure that multiple tasks can be processed simultaneously. This not only improves the efficiency of data processing, but also lays a solid foundation for subsequent model reasoning.
[0049] S2, the preprocessing module includes a video frame extraction unit, a text-to-speech unit, a facial data loading unit and a latent variable extraction unit. The video frame extraction unit extracts the character image video into a video frame image by using the multimedia processing tool FFmpeg (Fast Forward MPEG), and obtains the width, height, fps, channel and other information of the video.
[0050] The text-to-speech unit converts the text data uploaded by the user into audio data in .wav format through a text-to-speech model. The audio data is used in the inference stage of lip shape generation. The text-to-speech model can use a text-to-speech model based on deep learning.
[0051] The facial data loading unit is used for face detection, obtaining facial key points, lip area masks, facial latent variable representation and other facial related data.
[0052] Face detection: Use the face detection model in the DNN module of opencv to obtain the coordinates of the face detection frame (the coordinates of the upper left and lower right corners of the detection frame) and the width and height.
[0053] Face key point detection: Use the face key point detection model in Dlib to detect face key points in the face detection frame obtained in the previous step. A total of 68 feature points on the face can be marked, refer to Figure 6 , the specific distribution of these feature points is as follows:
[0054] contour:
[0055] Points 1-17: Along the contour of the face, from left ear to right ear.
[0056] Eyebrow:
[0057] Points 18-22: left eyebrow.
[0058] Points 23-27: right eyebrow.
[0059] nose:
[0060] Points 28-31: Bridge of the nose.
[0061] Points 32-36: around the nostrils.
[0062] Eye:
[0063] Points 37-42: around the left eye.
[0064] Points 43-48: around the right eye.
[0065] mouth:
[0066] Points 49-60: Outer lip outline.
[0067] Points 61-68: Inner lip contour.
[0068] The latent variable extraction unit obtains the latent variables of the face image, uses the encoder of the variational autoencoder unit to extract the face image obtained by the video frame extraction unit, uniformly scales the image to a 3-channel image with a width and height of 255, and obtains the latent variables of the face image through the encoder. The variational autoencoder unit includes two parts: the encoder and the decoder. The encoder compresses the input image into a probability distribution (mean and variance) in the latent space. The encoder consists of several convolutional layers and fully connected layers, which gradually extracts image features and maps them to the mean and variance of the latent space. According to the mean and variance output by the encoder, the reparameterization trick is used to sample latent variables from the latent space.
[0069] S3, the output data of the preprocessing module is input into the resource allocation module, and the resource allocation module allocates the data to the idle CUDA device according to the idle situation of the CUDA device resources. CUDA devices are hardware resources specifically used to execute model execution reasoning calculation tasks. The resource allocation module includes a resource management unit, a task ID allocation unit, and a resource recovery unit. The resource management unit uses a round robin algorithm to dynamically allocate all data and model resources. The model resources include face detection models, face key point detection models, variational autoencoders, and UNET models to idle CUDA devices. At the same time, the resource management unit monitors the idle and occupied situation of each CUDA device in real time. When the resource management unit allocates resources, it allocates data and model resources to idle CUDA devices according to the resource occupancy situation. When a new task is generated, the system will poll and select the next idle device from the available CUDA devices for allocation. For example, if there are multiple CUDA devices in the system, task 1 is allocated to CUDA device 1, task 2 is allocated to CUDA device 2, and so on. Whenever a task is completed, the resource recovery unit will release the occupied CUDA device resources and notify the resource management unit so that subsequent tasks can use them. The resource allocation module is also responsible for creating and maintaining a resource pool, which consists of a list of idle available resources of a CUDA device and a list of available model resources. The resource pool manages all available resources in the system, and the resource management unit allocates and recycles resources through the resource pool. Tasks are placed in the priority queue in the order in which the client requests arrive. The high-priority tasks that arrive first will obtain resources first, thereby ensuring that tasks can be processed in chronological order, which not only improves resource utilization, but also ensures the timely processing of high-priority tasks and enhances the overall performance of the system.
[0070] The task ID allocation unit allocates a unique streaming task ID to each data sending request from the client, which is used to identify the task and track its execution status and resource usage. Each task ID is unique in the system, ensuring that tasks are not confused or resource allocation conflicts occur. In addition, the task ID allocation unit records the creation time, execution status, and resource usage of the task, providing data support for subsequent resource management and optimization.
[0071] After the task is completed, the resource recycling unit adds the released idle CUDA device resources and model resources back to the resource pool maintained by the resource allocation module. It monitors the execution status of the streaming task and releases the unused resources in time to ensure that these resources can be reused efficiently and reasonably. The recycled resources are added back to the list of available resources for subsequent tasks to use. This resource recycling mechanism greatly improves the utilization of system resources and the flexibility of task processing.
[0072] S4, use the inference module to perform model inference on the preprocessed data to generate the character's lip shape consistent with the voice. Asynchronous processing and parallel computing are used to enable the system to process multiple inference tasks at the same time, improving the inference efficiency and response speed. The inference module includes an audio processing unit, a data alignment unit, a variational autoencoder unit, and a UNET unit. The specific steps are:
[0073] First, use the audio processing unit to extract audio feature data from the specified audio file path. By processing and extracting features from the audio, the original audio signal is converted into audio feature data for subsequent processing. The audio feature data contains the audio spectrum information, time domain information, Mel spectrum, speech content information, and phoneme timing information, which provides the basis for subsequent video frame and lip shape generation.
[0074] Next, the extracted audio feature data is combined with the latent variables of the video frame image and the face image to make the number of video frames consistent with the number of audio feature blocks to achieve audio and video alignment. If the length of the character video is shorter than the length of the audio, the video frame image is copied until it is the same length as the audio. Each latent variable of the face image corresponds to a video frame, so the latent variable of the face image is also copied to ensure the same length as the audio. This process ensures the synchronization of audio and video frames in time through the time series alignment algorithm, laying the foundation for generating lip-synced videos.
[0075] Then, according to the predefined batch size (i.e., the number of data processed each time, for example, the batch size is greater than or equal to 32), the batch of aligned audio features and latent variable data are obtained, and the obtained data is used for the reasoning task. The reasoning process uses multi-threading technology for parallel reasoning processing, making full use of the computing power of multi-core processors and GPUs to improve the efficiency and response speed of reasoning. During the reasoning process, the UNET unit and the variational autoencoder unit are processed synchronously. The UNET unit receives audio features and latent variables to generate latent variables of face images with consistent lip shape and audio. The characteristic of the UNET unit is that it can effectively extract and reconstruct image features through the encoder and decoder structure. The decoder of the same variational autoencoder unit is used to decode the latent variables and reconstruct an image with the same lip shape and audio data as the original image data. The size of the generated image is 255x255 pixels and the number of channels is 3.
[0076] After the image is generated, it is scaled back to the same size as the original one based on the size of the face detection box obtained by the face detection model in the previous preprocessing step. This step ensures that the generated image can be seamlessly integrated into the original video frame, maintaining image consistency and visual quality.
[0077] After the inference process is completed, the reconstructed video frames returned by each inference batch are processed. The reconstructed video frames are combined with the original video frames through the image synthesis unit to generate the final video frames with lip synchronization effects. In this step, the application of image synthesis technology ensures the consistency of color and details between the reconstructed frames and the original frames, improving the quality and visual effects of the video.
[0078] S5. Input the final video frame into the result output module. The result output module converts the final video frame into audio and video streams, and pushes them to the designated streaming media server through the RTMP protocol. Load balancing and caching technology are used to ensure that multiple streaming tasks can be carried out simultaneously, ensuring efficient and stable streaming without affecting system performance.
[0079] Specifically, the result output module consists of an audio and video synthesis unit, an RTMP streaming unit, a load balancing unit, and a cache unit. First, the audio and video synthesis unit uses FFmpeg to synthesize the uploaded audio file and the synthesized video frame with lip synchronization effect. FFmpeg is a multimedia processing tool that can efficiently perform audio and video synthesis operations to ensure the synchronization of audio and video. Through lip synchronization technology, the combination of video frames and audio is more natural, which improves the user's viewing experience.
[0080] Subsequently, the synthesized audio and video streams are pushed to the designated streaming media server through the RTMP streaming unit. RTMP (Real-Time Messaging Protocol) is a protocol designed specifically for audio and video streams, with the characteristics of low latency and high stability. The RTMP streaming unit pushes the synthesized audio and video streams to the streaming media server through the RTMP protocol, ensuring real-time and high-quality audio and video transmission. During the streaming process, the RTMP streaming unit is also responsible for handling network fluctuations to ensure the continuity and stability of the streaming.
[0081] In order to further optimize the streaming process, the result output module integrates a load balancing unit. This unit dynamically allocates CPU resources among multiple tasks to avoid individual tasks occupying too many resources, thereby ensuring overall performance. The load balancing unit adopts a round-robin algorithm. When executing the task of pushing video frames to the RTMP streaming server, the task is assigned to the idle and available CPU cores in sequence. Whenever a new task is generated, the system will select the next available CPU core to execute the task in turn. For example, if there are multiple CPU cores in the system, the first task will be assigned to CPU core 1, the second task will be assigned to CPU core 2, and so on, to ensure that the use of each CPU core resource is relatively balanced and avoid overloading a CPU core. Through this polling method, the load balancing unit can effectively manage the allocation of streaming resources for tasks, improve the overall system performance, and ensure that each streaming task has sufficient computing resource support.
[0082] The result output module also includes a cache unit, which is used to cache the final video frame results obtained by inference to improve the efficiency of streaming. During the execution of the streaming task, the cache unit temporarily stores the final video frame to reduce the delay of data processing. Through cache technology, the system can quickly respond to user requests and improve real-time performance. The cache unit can also intelligently manage the cache space according to task priority and resource usage to ensure priority processing of key tasks.
[0083] In some other preferred embodiments of the present invention, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in the above embodiment.
[0084] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0085] The above description is a detailed description of the preferred feasible embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modified changes completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.
Claims
1. A model reasoning method based on high-concurrency streaming, characterized in that: Includes steps: Receive streaming data from several clients; Preprocess the received streaming data to extract latent variables of character video frames, audio data, facial data, and facial images; Extract audio feature data from the preprocessed audio data, synthesize the audio feature data and the latent variables of the face image, generate an image that is consistent with the lip shape and audio data, and fuse the image with the original video frame of the streaming data to obtain the final video frame; Convert the final video frame into audio and video streams and push them to the designated streaming server via RTMP protocol, using load balancing and caching technology to perform multiple streaming tasks simultaneously; In the step of obtaining the final video frame, system resources are dynamically allocated and managed, a list of available CUDA devices and model resources is maintained, and a resource pool and priority queue mechanism are implemented.
2. According to claim 1, a model reasoning method based on high-concurrency streaming is characterized in that: The streaming data from multiple clients is received through the Flask framework. The Flask framework adopts the HTTP POST interface to receive audio, video and text data. A data parsing and verification unit is set in the Flask framework to perform preliminary parsing and verification on the received request data to ensure the integrity and correctness of the data.
3. According to claim 2, a model reasoning method based on high-concurrency streaming is characterized in that: The received streaming data is preprocessed by the preprocessing module, which includes a video frame extraction unit, a text-to-speech unit, a facial data loading unit and a latent variable extraction unit. The video frame extraction unit uses FFmpeg to extract the character image video into a video frame image. The text-to-speech unit converts the text data into audio data. The facial data loading unit is used to perform face detection and extract facial key points, lip area masks and facial latent variables to represent facial related data. The latent variable extraction unit is used to extract latent variables of the face image.
4. The model reasoning method based on high-concurrency streaming according to claim 1 is characterized in that: The resource allocation module is used to dynamically allocate and manage system resources. The resource allocation module includes a resource management unit. The resource management unit monitors the usage of each CUDA device in real time, dynamically adjusts the task allocation strategy on the CUDA device, and allocates model resources according to the size of the model and the priority of the task.
5. The model reasoning method based on high-concurrency streaming according to claim 4 is characterized in that: The resource allocation module also includes a task ID allocation unit, which allocates a unique task ID to each streaming task for identifying the task and tracking its execution status and resource usage.
6. According to claim 4, a model reasoning method based on high-concurrency streaming is characterized in that: The resource allocation module also includes a resource recovery unit, which adds the released resources back into the available resource list after the current streaming task is completed.
7. The model reasoning method based on high-concurrency streaming according to claim 1 is characterized in that: The inference module processes the preprocessed audio data to obtain the final video frame. The inference module includes an audio processing unit, a data alignment unit, and a data generator. The audio processing unit is used to extract audio feature data from the audio data. The data alignment unit aligns the audio feature data with the video frame data. The data generator obtains the aligned audio features and latent variables.
8. The model reasoning method based on high-concurrency streaming according to claim 7 is characterized in that: The inference module also includes a variational autoencoder unit, a UNET unit and a synthesis unit. The UNET unit receives the audio features and latent variables output by the data generator to generate latent variables of a face image with consistent lip shape and audio. The variational autoencoder unit decodes the latent variables of the face image and reconstructs an image with the same lip shape and audio data as the original image data size. The synthesis unit combines the image with the original video frame to generate a final video frame with a lip synchronization effect.
9. A model reasoning system based on high-concurrency streaming, used to implement a model reasoning method based on high-concurrency streaming as described in any one of claims 1 to 8, characterized in that: It includes a data receiving module, a preprocessing module, a resource allocation module, a reasoning module and a result output module; The preprocessing module includes a video frame extraction unit, a text-to-speech unit, a facial data loading unit, and a latent variable extraction unit; The resource allocation module includes a resource management unit, a task ID allocation unit and a resource recovery unit; The inference module includes an audio processing unit, a data alignment unit, a data generator, a variational autoencoder unit, a UNET unit and a synthesis unit; The result output module is composed of an audio and video synthesis unit, an RTMP streaming unit, a load balancing unit and a cache unit.
10. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor executes the steps of a model reasoning method based on high-concurrency streaming as described in any one of claims 1 to 8.