AI model acceleration method and acceleration system for ultra-high-definition 8k video
By pre-training the AI model, pruning optimization, time domain and spatial domain pruning processing, and dynamically adjusting the pruning strategy according to the content complexity, the calculation complexity and limited resources of the AI model when processing ultra-high-definition 8K videos are solved. The problem is to achieve efficient and real-time video processing.
Patent Information
- Application Number
- CN202411308552.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-19
AI Technical Summary
Existing AI models face the problems of excessive computational complexity and limited resources when processing ultra-high-definition 8K videos, which are difficult to meet the needs of real-time and efficientness.
By comprehensively applying pre-trained models, pruning optimization, video data time and spatial domain pruning processing, and dynamic pruning strategy adjustment based on content complexity, the AI model is optimized to achieve efficient processing of ultra-high-definition 8K videos. Specific steps include selecting pre-trained models for redundant pruning optimization, time-domain and spatial domain pruning processing, model parameters and configuration adjustment, and dynamically adjusting pruning strategies according to content complexity.
It realizes efficient processing of ultra-high-definition 8K video, meets the needs of real-time and accuracy, significantly improves processing speed and efficiency, and reduces computing complexity and resource consumption.
Smart Images

Figure CN118823654B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of AI model technology, and in particular to an AI model acceleration method and acceleration system applied to ultra-high-definition 8k video. Background Art
[0002] With the rapid development of ultra-high-definition video technology, especially the popularity of 8K video content, the performance requirements for video processing systems are increasing. Ultra-high-definition 8K video, with its extremely high resolution and rich details, brings users an unprecedented visual experience, but it also poses huge challenges to various links such as video processing, transmission and storage. When processing ultra-high-definition 8K videos, traditional video processing methods and models often face problems such as large computational complexity, slow processing speed, and high resource consumption, making it difficult to meet the requirements of real-time and high efficiency.
[0003] To meet these challenges, artificial intelligence (AI) technology has been widely used in the field of video processing, especially in model acceleration. However, existing AI models also face bottlenecks such as high model complexity and limited computing resources when processing ultra-high-definition 8K videos. Therefore, how to optimize AI models and improve their efficiency and accuracy in processing ultra-high-definition 8K videos has become a hot topic and difficulty in current research.
[0004] In the prior art, a common optimization method is to prune the pre-trained model to reduce redundant elements in the model, reduce computational complexity and resource consumption. However, this method often requires a trade-off between model accuracy and computational efficiency, and the optimization effect may vary for different video content.
[0005] Therefore, it is necessary to provide an AI model acceleration method and acceleration system for ultra-high-definition 8k video to solve the above technical problems. Summary of the invention
[0006] In order to solve the above technical problems, the present invention provides an AI model acceleration method and acceleration system applied to ultra-high-definition 8K video. Through the comprehensive application of pre-trained model pruning optimization, video data time domain and spatial domain pruning processing, and dynamic pruning strategy adjustment based on content complexity, efficient processing of ultra-high-definition 8K video is achieved, meeting the requirements of real-time and accuracy.
[0007] The present invention provides an AI model acceleration method applied to ultra-high-definition 8k video, and the acceleration method comprises the following steps:
[0008] S1: Selecting a pre-trained model as a basic model of the pre-processed raw video data, and pruning and optimizing the basic model by diagnosing redundancy;
[0009] S2: Performing time domain pruning processing on the original video data to obtain first video data;
[0010] S3: Performing spatial domain pruning processing on the first video data to obtain second video data;
[0011] S4: adjusting a basic model based on the first video data and the second video data, wherein the adjustment includes model parameter adjustment and model configuration adjustment;
[0012] S5: Dynamically change the pruning strategy of the adjusted basic model according to the content complexity of the second video data to achieve accelerated processing of the basic model.
[0013] Preferably, step S1 comprises the following steps:
[0014] S11: Select a pre-trained model that matches the ultra-high-definition 8k video processing task from the preset model library as the basic model:
[0015] S12: Evaluate the basic model and identify redundant elements in the basic model, wherein the redundant elements include redundant connections, redundant layers, and redundant parameters:
[0016] S13: Pruning the basic model based on the identified redundant elements, and retraining the pruned basic model.
[0017] Preferably, step S2 comprises the following steps:
[0018] S21: Identify redundant information between video frames in the original video data based on the time sequence:
[0019] S22: Designing a time domain pruning strategy according to the redundant information, wherein the time domain pruning strategy includes one or more combinations of frame skipping, frame content prediction, and key frame extraction:
[0020] S23: Apply the time-domain pruning strategy to process the original video data to generate the first video data after time-domain pruning.
[0021] Preferably, step S3 comprises the following steps:
[0022] S31: Perform spatial domain feature analysis on each frame of the first video data to identify redundant regions in each frame:
[0023] S32: Designing a spatial domain pruning strategy according to the redundant region, wherein the spatial domain pruning strategy includes reducing the resolution of the redundant region and merging regions, or a combination of the two:
[0024] S33: Apply the spatial domain pruning strategy to each frame of the first video data, and generate second video data after spatial domain pruning by reducing the computational complexity and data volume of the redundant area.
[0025] Preferably, step S4 comprises the following steps:
[0026] S41: Analyze the feature distribution in the first video data and the second video data:
[0027] S42: Adjusting the parameters and configuration of the basic model based on the characteristic distribution:
[0028] S43: running the second video data on the adjusted basic model, while recording comparison data between the output result of the basic model and the predetermined performance index, and finally iteratively adjusting the basic model based on the comparison data until a predetermined performance optimization target is achieved.
[0029] Preferably, step S5 comprises the following steps:
[0030] S51: Using a preset complexity evaluation algorithm, perform content complexity evaluation on each frame of the second video data to obtain content complexity:
[0031] S52: Divide the complexity level according to the span of the content complexity, and predefine the corresponding pruning strength for each complexity level:
[0032] S53: adjusting the pruning ratio of the basic model in real time according to the pruning intensity to ensure a balance between the processing speed and accuracy of the basic model.
[0033] The present invention also provides an AI model acceleration system applied to ultra-high-definition 8k video, which is used to execute an AI model acceleration method applied to ultra-high-definition 8k video. The acceleration system includes:
[0034] A basic model preparation module is used to select a pre-trained model as a basic model for the pre-processed raw video data, and prune and optimize the basic model by diagnosing redundancy;
[0035] A time domain pruning module, used for performing time domain pruning processing on the original video data to obtain first video data;
[0036] A spatial domain pruning module, configured to perform spatial domain pruning processing on the first video data to obtain second video data;
[0037] A model adjustment module, configured to adjust a basic model based on the first video data and the second video data, wherein the adjustment includes model parameter adjustment and model configuration adjustment;
[0038] A dynamic pruning strategy module is used to dynamically change the pruning strategy of the adjusted basic model according to the content complexity of the second video data to achieve accelerated processing of the basic model. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A flowchart of an AI model acceleration method for ultra-high-definition 8k video provided by the present invention;
[0040] Figure 2 This is a module structure diagram of an AI model acceleration method applied to ultra-high-definition 8k video provided by the present invention. DETAILED DESCRIPTION
[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.
[0042] It should also be noted that, for ease of description, only the parts related to the present invention, but not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc.
[0043] Embodiment 1
[0044] The present invention provides an AI model acceleration method for ultra-high-definition 8k video. Figure 1 As shown, the acceleration method includes the following steps:
[0045] S1: Select a pre-trained model as a basic model of the pre-processed original video data, and prune and optimize the basic model through diagnostic redundancy.
[0046] Specifically, step S1 includes the following steps:
[0047] S11: Select a pre-trained model that matches the ultra-high-definition 8k video processing task from the preset model library as the basic model.
[0048] In this embodiment, a pre-trained model suitable for ultra-high-definition 8K video processing tasks is screened out from a preset model library, wherein the pre-trained model library includes a variety of trained models, such as ResNet, EfficientNet, etc.; then, according to the performance indicators of the model (including but not limited to accuracy and processing speed) and the complexity of the model, the model that best suits the current task requirements is selected as the basic model; since the selected pre-trained model has been pre-trained on a large-scale data set, it can be ensured that it has good generalization ability and basic processing capabilities.
[0049] This step can greatly reduce the time and resources required to train the model from scratch by selecting a pre-trained model that matches the ultra-high-definition 8K video processing task. At the same time, the pre-trained model has been fully learned on a large-scale dataset and can provide good initial performance, laying a good foundation for subsequent pruning and optimization.
[0050] S12: Evaluate the basic model and identify redundant elements in the basic model, wherein the redundant elements include redundant connections, redundant layers and redundant parameters.
[0051] In this embodiment, model diagnostic tools, including but not limited to TensorFlow Profiler and PyTorchProfiler, are used to evaluate the basic model; redundant connections, redundant layers, and redundant parameters in the model are identified. These redundant elements refer to parts that do not play an important role or contribute less in the model.
[0052] Use visualization tools (including but not limited to TensorBoard) to help understand the bottlenecks of the model and determine which parts can be safely pruned without affecting overall performance.
[0053] This step can effectively identify the redundant parts in the model by evaluating the basic model, thus providing a basis for subsequent pruning work. By removing redundant elements, the complexity of the model can be significantly reduced and the processing speed of the model can be improved.
[0054] S13: Pruning the basic model based on the identified redundant elements, and retraining the pruned basic model.
[0055] In this embodiment, the basic model is pruned according to the identified redundant elements, including but not limited to deleting redundant connections, redundant layers and redundant parameters; then, the pruned model is retrained to recover the performance degradation caused by pruning. A smaller learning rate is used during the retraining process, and the model parameters are gradually adjusted. After the retraining is completed, the model is verified to ensure that its performance is not lower than the original model.
[0056] This step can further optimize the model structure and reduce the amount of calculation through pruning and retraining, while maintaining the processing accuracy of the model. The pruned model can complete the task faster when processing ultra-high-definition 8K video, thereby improving the response speed and efficiency of the overall system.
[0057] S2: Performing time domain pruning processing on the original video data to obtain first video data.
[0058] Specifically, step S2 includes the following steps:
[0059] S21: Identify redundant information between video frames in the original video data based on the time series.
[0060] In this embodiment, the time series analysis tool OpenCV is used to analyze each frame in the original video data to identify the similarities and differences between consecutive frames, especially the position, shape and motion trajectory of the target object. By calculating the optical flow field and feature point matching between adjacent frames, the redundant information between the video frames, that is, the frames with higher similarity, is identified.
[0061] This step can identify the redundant information between video frames to determine which frames have smaller changes, thereby providing a basis for the subsequent time domain pruning strategy. At the same time, by reducing the processing of redundant frames, the computational complexity of the model can be significantly reduced and the processing speed can be improved.
[0062] S22: Designing a time domain pruning strategy according to the redundant information, wherein the time domain pruning strategy includes one or more combinations of frame skipping, frame content prediction, and key frame extraction.
[0063] In this embodiment, based on the identified redundant information, a time domain pruning strategy is designed, including but not limited to frame skipping, frame content prediction and key frame extraction, wherein:
[0064] Frame skipping means that for frames with high similarity, some frames can be selectively skipped to reduce the number of processing times.
[0065] Frame content prediction uses the processed key frames to predict the content of the intermediate frames, reducing unnecessary feature extraction.
[0066] Key frame extraction is to select frames that are representative or contain important information as key frames for detailed processing.
[0067] This step can further reduce unnecessary calculations and improve the speed of the model in processing videos by designing a reasonable time domain pruning strategy. At the same time, key frame extraction can ensure that the model captures the most important information when processing videos, while frame skipping and frame content prediction can significantly reduce the amount of calculation.
[0068] S23: Apply the time-domain pruning strategy to process the original video data to generate the first video data after time-domain pruning.
[0069] In this embodiment, the designed time domain pruning strategy is applied to process the original video data.
[0070] For skipped frames, the features of key frames can be directly used for fast matching or prediction.
[0071] For key frames, detailed feature extraction and processing are performed.
[0072] Generate first video data after time domain pruning, the data including a frame sequence after pruning.
[0073] This step uses time domain pruning to generate the first video data, which not only reduces the amount of calculation but also ensures the retention of key information. At the same time, the processed video data is lighter, which can significantly improve the speed and efficiency of the model in processing ultra-high-definition 8K videos.
[0074] S3: Perform spatial domain pruning processing on the first video data to obtain second video data.
[0075] Specifically, step S3 includes the following steps:
[0076] S31: Perform spatial domain feature analysis on each frame of the first video data to identify redundant areas in each frame.
[0077] In this embodiment, a target detection algorithm (including but not limited to YOLO and Faster R-CNN) is used to perform spatial domain feature analysis on each frame of the first video data to identify important areas (such as areas where the target object is located) and redundant areas (such as backgrounds or areas with little change) in each frame; and through feature map analysis and target detection results, it is determined which areas can be simplified or ignored during the processing.
[0078] This step can identify important target areas and redundant areas by performing spatial domain feature analysis on each frame, thereby providing a basis for subsequent spatial domain pruning strategies; at the same time, the identified redundant areas can be simplified to reduce the amount of calculation and improve the processing speed.
[0079] S32: Designing a spatial domain pruning strategy according to the redundant region, wherein the spatial domain pruning strategy includes reducing the resolution of the redundant region and merging regions, or a combination of the two.
[0080] In this embodiment, based on the identified redundant regions, a spatial domain pruning strategy is designed, including but not limited to:
[0081] Reduce the resolution of redundant areas: For redundant areas, reduce their resolution to reduce the amount of calculation.
[0082] Region merging: Merge adjacent redundant regions into one large region for simplification.
[0083] Combining the above two strategies, the redundant areas are processed to reduce the computational complexity and data volume.
[0084] This step can significantly reduce the amount of calculation and increase the speed at which the model processes videos by reducing the resolution of redundant areas and performing region merging. At the same time, these strategies ensure that the model can focus on important target areas when processing videos and ignore redundant areas, thereby improving processing efficiency.
[0085] S33: Apply the spatial domain pruning strategy to each frame of the first video data, and generate second video data after spatial domain pruning by reducing the computational complexity and data volume of the redundant area.
[0086] In this embodiment, a designed spatial domain pruning strategy is applied to process each frame of the first video data.
[0087] For redundant regions, reduce their resolution or merge them into larger regions for simplification.
[0088] For important areas, their original resolution is maintained and detailed feature extraction is performed.
[0089] Generate second video data after spatial domain pruning, the data including a frame sequence after pruned processing.
[0090] This step uses spatial domain pruning to generate the second video data, which not only reduces the amount of calculation but also ensures the retention of important information. At the same time, the processed video data is lighter, which can significantly improve the speed and efficiency of the model in processing ultra-high-definition 8K videos.
[0091] S4: Based on the first video data and the second video data, adjusting the basic model, wherein the adjustment includes model parameter adjustment and model configuration adjustment.
[0092] Specifically, step S4 includes the following steps:
[0093] S41: Analyze the feature distribution in the first video data and the second video data.
[0094] In this embodiment, feature analysis tools (including but not limited to statistical analysis functions in TensorFlow and PyTorch) are used to perform feature distribution analysis on the first video data and the second video data.
[0095] Analyze the feature distribution, including but not limited to the importance, distribution range, statistical characteristics, etc. of the feature. Through statistical analysis, identify the main patterns and outliers in the feature distribution for subsequent adjustment of the model.
[0096] This step can better understand the characteristics and patterns of the data by performing feature distribution analysis on the first video data and the second video data, providing a basis for subsequent model adjustments. At the same time, feature distribution analysis helps to identify which features are critical to the performance of the model, thereby guiding the adjustment of model parameters and configurations.
[0097] S42: Adjusting the parameters and configuration of the basic model based on the feature distribution.
[0098] In this embodiment, the parameters of the basic model, such as weights, bias terms, etc., are adjusted according to the feature distribution.
[0099] Modify the model configuration, such as adjusting the network structure, number of layers, activation function, etc.
[0100] At the same time, you can use the automatic parameter tuning tool Grid Search to find the optimal parameter combination.
[0101] This step can further optimize the performance of the model by adjusting the model parameters and configuration; at the same time, reasonable parameter and configuration adjustments can improve the accuracy and efficiency of the model when processing video data.
[0102] More specifically:
[0103] The specific learning rate adjustment in parameter adjustment is:
[0104] The learning rate directly affects the convergence speed and stability of the model. The feature distribution can help us understand the complexity and diversity of the data and thus choose an appropriate learning rate.
[0105] If the feature distribution shows that the data is more complex and diverse, a smaller learning rate can be used, because a smaller learning rate can help the model explore the minimum value of the loss function more carefully.
[0106] If the feature distribution shows that the data is relatively simple, a larger learning rate can be used to speed up convergence.
[0107] By adjusting the learning rate based on the feature distribution, we can ensure that the model does not oscillate when processing complex data and converges faster when processing simple data.
[0108] The regularization coefficient adjustment in parameter adjustment is specifically:
[0109] The regularization coefficient is used to control the model complexity and prevent overfitting, and the feature distribution can reveal the noise level and complexity in the data.
[0110] If the feature distribution shows more noise or higher complexity in the data, the regularization coefficient can be increased to reduce the risk of overfitting.
[0111] If the feature distribution shows that the data is cleaner or less complex, the regularization coefficient can be reduced to allow the model to better fit the data.
[0112] By adjusting the regularization coefficient according to the feature distribution, the fitting ability and generalization ability of the model can be balanced to avoid overfitting or underfitting.
[0113] The network architecture adjustments in the configuration adjustment are as follows:
[0114] The choice of network architecture depends on the feature distribution of the data, which can determine whether a deeper or wider network is needed.
[0115] If the feature distribution shows high data complexity, the depth or width of the network can be increased to enhance the expressiveness of the model.
[0116] If the feature distribution shows that the data is relatively simple, the depth or width of the network can be reduced to reduce the amount of computation.
[0117] By adjusting the network architecture based on feature distribution, we can reduce the demand for computing resources and improve processing speed while ensuring model performance.
[0118] The activation function selection in the configuration adjustment is as follows:
[0119] The choice of activation function depends on the characteristic distribution of the data. Different activation functions have different effects on the learning ability and expression ability of the model.
[0120] If the feature distribution shows that there are more nonlinear relationships in the data, you can use nonlinear activation functions such as ReLU, Leaky ReLU, or Swish to enhance the nonlinear expression ability of the model.
[0121] If the feature distribution shows that the data is relatively linear, you can consider using a linear activation function or a simple nonlinear activation function to simplify the model.
[0122] By selecting the appropriate activation function based on the feature distribution, you can improve the gradient flow of the model, speed up training, and improve the performance of the model.
[0123] The batch size adjustment in the configuration adjustment is as follows:
[0124] The choice of batch size depends on the feature distribution of the data, which is used to assist in determining the appropriate batch size.
[0125] If the feature distribution shows that the data is more complex, a smaller batch size can be used to better fit the data.
[0126] If the feature distribution shows that the data is simple, a larger batch size can be used to speed up training.
[0127] By adjusting the batch size based on feature distribution, you can improve training speed and efficiency while ensuring model performance.
[0128] For example, assume that feature distribution analysis shows that there are a large number of moving objects and complex backgrounds in the data:
[0129] Learning rate adjustment: Choose a smaller learning rate (such as 0.001) to ensure that the model can explore the minimum value of the loss function more carefully and avoid oscillation in complex data.
[0130] Regularization coefficient adjustment: Choose a larger regularization coefficient (such as 0.001) to reduce the risk of overfitting and ensure the generalization ability of the model on complex data.
[0131] Network architecture adjustment: Choose a deeper network architecture (such as ResNet-101) to enhance the model's ability to express complex features.
[0132] Activation function selection: Select a nonlinear activation function such as Leaky ReLU to enhance the nonlinear expression ability of the model and better handle complex data.
[0133] Batch size tuning: Choose a smaller batch size (such as 16) to better fit complex data
[0134] S43: running the second video data on the adjusted basic model, while recording comparison data between the output result of the basic model and the predetermined performance index, and finally iteratively adjusting the basic model based on the comparison data until a predetermined performance optimization target is achieved.
[0135] In this embodiment, the second video data is run on the adjusted basic model, the output results of the model are recorded, and compared with predetermined performance indicators (such as accuracy, processing speed, etc.); if the performance of the model does not reach the predetermined target, the model is iteratively adjusted according to the comparison data until the predetermined performance optimization target is reached.
[0136] By iteratively adjusting the model, we can ensure that the model can achieve the expected performance standards when processing the second video data. At the same time, the iterative adjustment process can continuously optimize the model, making it more efficient and accurate when processing ultra-high-definition 8K video.
[0137] S5: Dynamically change the pruning strategy of the adjusted basic model according to the content complexity of the second video data to achieve accelerated processing of the basic model.
[0138] Specifically, step S5 includes the following steps:
[0139] S51: Using a preset complexity evaluation algorithm, perform content complexity evaluation on each frame of the second video data to obtain content complexity.
[0140] In this embodiment, an image complexity evaluation algorithm (including but not limited to calculating the number of edges and texture complexity of each frame) is used to perform content complexity evaluation on each frame in the second video data. The evaluation algorithm can quantify the complexity of each frame based on image processing technology (including but not limited to edge detection and feature point detection), and obtain a comprehensive complexity score by calculating the feature values of each frame (including but not limited to the number of edges and the degree of color change).
[0141] Exemplarily, the edge detection algorithm Canny may be used to evaluate the number of edges in each frame, and a greater number of edges indicates a higher complexity.
[0142] By evaluating the complexity of each frame, we can determine which frames have more complex contents and which are simpler. At the same time, the complexity evaluation results provide a basis for subsequent dynamic pruning strategies, which helps to adopt different processing methods between video frames of different complexities.
[0143] S52: Divide the complexity levels according to the span of the content complexity, and predefine a corresponding pruning strength for each complexity level.
[0144] In this embodiment, the complexity score is divided into multiple levels, specifically low complexity, medium complexity, and high complexity, and a corresponding pruning intensity is predefined for each complexity level, for example:
[0145] Low-complexity frames can adopt stronger pruning strategies, such as drastically reducing the resolution or drastically pruning the model.
[0146] Moderately complex frames can adopt moderately intensive pruning strategies, such as moderately reducing the resolution or moderately pruning the model.
[0147] High complexity frames can use weaker pruning strategies, such as only slightly reducing the resolution or slightly pruning the model.
[0148] In this way, the pruning strength can be flexibly adjusted according to the complexity of each frame. By dividing the complexity levels and predefining the pruning strength for each level, it can be ensured that the model adopts different strategies when processing video frames of different complexities, thereby improving the processing speed while ensuring accuracy.
[0149] S53: adjusting the pruning ratio of the basic model in real time according to the pruning intensity to ensure a balance between the processing speed and accuracy of the basic model.
[0150] In this embodiment, when processing each frame, the pruning ratio of the basic model is adjusted in real time according to the complexity level of the frame.
[0151] For low-complexity frames, a stronger pruning strategy is adopted to reduce the amount of computation.
[0152] For high-complexity frames, a weaker pruning strategy is adopted to ensure the processing accuracy of the model.
[0153] By dynamically adjusting the pruning ratio, we can ensure that the model achieves the best performance balance when processing each frame.
[0154] By dynamically adjusting the pruning ratio, it can be ensured that the model can maintain a high processing speed and ensure processing accuracy when processing video frames of different complexities; at the same time, the dynamic pruning strategy enables the model to be more efficient when processing ultra-high-definition 8K videos, while ensuring the retention of key information and improving the response speed and efficiency of the overall system.
[0155] Exemplarily, the complexity levels and pruning strategies are predefined:
[0156] Low complexity frames (such as static backgrounds):
[0157] Pruning ratio: 70% (i.e., model parameters are reduced by 70%)
[0158] Processing strategy: reduce the resolution and the number of feature extraction layers.
[0159] Medium complexity frames (such as a small number of moving objects):
[0160] Pruning ratio: 40%
[0161] Processing strategy: moderately reduce the resolution and the number of feature extraction layers.
[0162] High complexity frames (such as multiple moving objects, complex background):
[0163] Pruning ratio: 10%
[0164] Processing strategy: almost no reduction in resolution and almost no reduction in the number of feature extraction layers.
[0165] Suppose you are processing a video that contains a largely static background and occasionally a few moving objects.
[0166] When encountering low-complexity frames (such as a background that remains static for a long time), the model will apply a 70% pruning ratio, which means that the model will significantly reduce the amount of calculation, thereby increasing processing speed.
[0167] When encountering frames of medium complexity (such as the background remains unchanged but several moving objects appear), the model will adjust to a 40% pruning ratio to ensure that a certain accuracy is maintained when processing these frames.
[0168] When encountering high-complexity frames (such as those with complex backgrounds and multiple moving objects), the model will adjust to a 10% pruning ratio to ensure that all details are correctly captured, thereby maintaining the processing accuracy of the model.
[0169] The working principle of an AI model acceleration method for ultra-high-definition 8k video provided by the present invention is as follows:
[0170] First, a pre-trained model that matches the UHD 8K video processing task is selected from the preset model library as the base model. This base model already has certain video processing capabilities, but may contain a large number of redundant connections, redundant layers, and redundant parameters, resulting in low computational efficiency. Therefore, the complexity of the model is reduced by evaluating the redundant elements in the model and using pruning technology to remove these redundant parts. The pruned model is retrained to restore or improve its performance when processing UHD videos.
[0171] After obtaining the optimized basic model, the original ultra-high-definition 8K video data is pruned in the time domain. This step identifies redundant information between frames by analyzing the time series relationship between video frames. Based on this information, time domain pruning strategies such as frame skipping, frame content prediction, and key frame extraction are designed and applied to reduce the temporal redundancy of video data. The video data (first video data) after time domain pruning retains key information while significantly reducing the amount of data, which reduces the burden on subsequent processing.
[0172] In order to further reduce the computational complexity and data volume of the video data, the first video data is pruned in the spatial domain. This step identifies the redundant regions in the frame by performing spatial domain feature analysis on each part of the video frame. Then, a spatial domain pruning strategy is designed based on the characteristics of the redundant regions, such as reducing the resolution of the redundant regions or merging the regions. The application of these strategies reduces the computational complexity of the redundant regions and generates the second video data after spatial domain pruning.
[0173] Based on the feature distribution of the first video data and the second video data, the parameters and configuration of the basic model are adjusted. This step optimizes the parameter settings and internal configuration of the model by analyzing the feature changes in the video data, so that the model is more suitable for processing the pruned video data. By running the adjusted model and recording the comparison data of the output results and the predetermined performance indicators, iterative adjustments are made until the model reaches the predetermined performance optimization target.
[0174] Finally, the pruning strategy of the adjusted basic model is dynamically changed according to the content complexity of the second video data. This step evaluates the content complexity of each frame of the video data through a preset complexity evaluation algorithm, and divides the complexity level according to the evaluation results. The corresponding pruning intensity is predefined for each complexity level, and the pruning ratio is adjusted in real time during the model processing process. This dynamic adjustment strategy ensures that the model can maintain a balance between processing speed and accuracy when processing video frames of different complexities, thereby achieving efficient processing of ultra-high-definition 8K videos.
[0175] In summary, the working principle of the present invention realizes efficient, flexible and adaptive processing of ultra-high-definition 8K video by comprehensively using pre-trained model pruning optimization, video data time domain and spatial domain pruning processing, and dynamic pruning strategy adjustment based on content complexity.
[0176] Embodiment 2
[0177] The present invention also provides an AI model acceleration system applied to ultra-high-definition 8k video, which is used to execute an AI model acceleration method applied to ultra-high-definition 8k video, referring to Figure 2 As shown, the acceleration system includes:
[0178] The basic model preparation module 100 is used to select a pre-trained model as a basic model of the pre-processed original video data, and prune and optimize the basic model by diagnosing redundancy.
[0179] The time domain pruning module 200 is used to perform time domain pruning processing on the original video data to obtain first video data.
[0180] The spatial domain pruning module 300 is used to perform spatial domain pruning processing on the first video data to obtain second video data.
[0181] The model adjustment module 400 is used to adjust the basic model based on the first video data and the second video data, wherein the adjustment includes model parameter adjustment and model configuration adjustment.
[0182] The dynamic pruning strategy module 500 is used to dynamically change the pruning strategy of the adjusted basic model according to the content complexity of the second video data to achieve accelerated processing of the basic model.
[0183] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0184] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.
[0185] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or includes elements that are inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.
Claims
1. An AI model acceleration method applied to ultra-high-definition 8k video, characterized in that: The acceleration method includes the following steps: S1: Selecting a pre-trained model as a basic model of the pre-processed raw video data, and pruning and optimizing the basic model by diagnosing redundancy; S2: Performing time domain pruning processing on the original video data to obtain first video data; S3: Performing spatial domain pruning processing on the first video data to obtain second video data; S4: adjusting a basic model based on the first video data and the second video data, wherein the adjustment includes model parameter adjustment and model configuration adjustment; S5: dynamically changing the pruning strategy of the adjusted basic model according to the content complexity of the second video data to achieve accelerated processing of the basic model; Step S2 includes the following steps: S21: Identify redundant information between video frames in the original video data based on the time sequence: S22: Designing a time domain pruning strategy according to the redundant information, wherein the time domain pruning strategy includes one or more combinations of frame skipping, frame content prediction, and key frame extraction: S23: Apply the time domain pruning strategy to process the original video data to generate the first video data after time domain pruning; Step S3 includes the following steps: S31: Perform spatial domain feature analysis on each frame of the first video data to identify redundant regions in each frame: S32: Designing a spatial domain pruning strategy according to the redundant region, wherein the spatial domain pruning strategy includes reducing the resolution of the redundant region and merging regions, or a combination of the two: S33: applying the spatial domain pruning strategy to each frame of the first video data, and generating second video data after spatial domain pruning by reducing the computational complexity and data volume of the redundant region; Step S5 includes the following steps: S51: Using a preset complexity evaluation algorithm, perform content complexity evaluation on each frame of the second video data to obtain content complexity: S52: Divide the complexity level according to the span of the content complexity, and predefine the corresponding pruning strength for each complexity level: S53: adjusting the pruning ratio of the basic model in real time according to the pruning intensity to ensure a balance between the processing speed and accuracy of the basic model.
2. According to claim 1, the AI model acceleration method applied to ultra-high-definition 8k video is characterized in that: Step S1 includes the following steps: S11: Select a pre-trained model that matches the ultra-high-definition 8k video processing task from the preset model library as the basic model: S12: Evaluate the basic model and identify redundant elements in the basic model, wherein the redundant elements include redundant connections, redundant layers, and redundant parameters: S13: Pruning the basic model based on the identified redundant elements, and retraining the pruned basic model.
3. The AI model acceleration method for ultra-high-definition 8k video according to claim 1, characterized in that: Step S4 includes the following steps: S41: Analyze the feature distribution in the first video data and the second video data: S42: Adjusting the parameters and configuration of the basic model based on the characteristic distribution: S43: running the second video data on the adjusted basic model, while recording comparison data between the output result of the basic model and the predetermined performance index, and finally iteratively adjusting the basic model based on the comparison data until a predetermined performance optimization target is achieved.
4. An AI model acceleration system for ultra-high-definition 8k video, used to execute an AI model acceleration method for ultra-high-definition 8k video as described in any one of claims 1 to 3, characterized in that: The acceleration system includes: A basic model preparation module is used to select a pre-trained model as a basic model for the pre-processed raw video data, and prune and optimize the basic model by diagnosing redundancy; A time domain pruning module, used for performing time domain pruning processing on the original video data to obtain first video data; A spatial domain pruning module, configured to perform spatial domain pruning processing on the first video data to obtain second video data; A model adjustment module, configured to adjust a basic model based on the first video data and the second video data, wherein the adjustment includes model parameter adjustment and model configuration adjustment; A dynamic pruning strategy module is used to dynamically change the pruning strategy of the adjusted basic model according to the content complexity of the second video data to achieve accelerated processing of the basic model.
