Optimization method and system for improving visual effect

Abstract

The invention discloses an optimization method and system for improving a visual effect, and the method comprises the steps: S1, a client preferentially starts an asynchronous monitoring thread, monitors the resource use conditions of a CPU and a GPU of the client within a certain period of time, and divides the resources into a high grade, a middle grade and a low grade according to the remaining conditions of the computing resources; the method comprises the following steps: S1, carrying out frame compensation on a video stream, S2, integrating a frame compensation algorithm toolkit which can realize a frame compensation effect on a part of frame pictures in the video stream by a client, and S3, when the video stream is high-grade, carrying out frame compensation operation by adopting a multi-quantity interval frame extraction mode and adopting the toolkit, and carrying out calculation optimization on the rest of frames according to an old mode; when the grade is a middle grade, a middle-quantity interval frame extraction mode is adopted, and a toolkit is adopted for frame supplement operation; and when the frame is in a low grade, frame compensation operation is carried out by adopting a toolkit according to a mode of extracting frames at a small interval. According to the invention, frame compensation operation of different video special effects can be carried out according to configuration of different devices, and the user experience of the system is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer communication, and in particular to an optimization method and system for improving visual effects. Background Art

[0002] There are various client apps that can be used to view videos, pictures, and other functions. The user's client device may have various scenarios with different performances. Usually, various video streams have multiple resolutions, and different resolutions are used to display video content. However, in the same resolution process, for different client models, the display of video special effects is usually what the server-side video generates, and it cannot be adjusted again. Or integrate a small algorithm on the client to convert each frame of the special effect video to achieve better results. However, different client models and devices will affect the final effect. Summary of the invention

[0003] In order to overcome the above problems, the purpose of the present invention is to provide an optimization method and system for improving visual effects, which can realize frame interpolation operations of different video special effects according to the configuration of different devices, thereby improving the user experience of the system.

[0004] The present invention is implemented by the following scheme: an optimization method for improving visual effects, the method comprising the following steps:

[0005] Step S1, the client first starts an asynchronous monitoring thread, which mainly monitors the computing power of the client device for a period of time, monitors the resource usage of the client CPU and GPU within a period of time, and divides it into three levels of high, medium and low according to the remaining computing resources;

[0006] Step S2: The client integrates a frame interpolation algorithm toolkit, which can achieve the effect of interpolating some frames in the video stream, using the existing content of the previous frame to generate the content of this frame and directly replace it;

[0007] Step S3, when the remaining resources are of high quality, a method of extracting frames at large intervals is adopted, and a toolkit is used to perform frame supplementation operations, and the remaining frames are still calculated and optimized according to the old method; when the remaining resources are of medium quality, a method of extracting frames at medium intervals is adopted, and a toolkit is used to perform frame supplementation operations; when the remaining resources are of low quality, a method of extracting frames at small intervals is adopted, and a toolkit is used to perform frame supplementation operations.

[0008] Furthermore, the remaining computing resources in step S1 are divided into three levels: high, medium and low. When the remaining resources are greater than 60%, it is a high level; when the remaining resources are between 30% and 60%, it is a medium level; and when the remaining resources are less than 30%, it is a low level.

[0009] Furthermore, the content of the frame generated in step S2 is directly replaced by: based on an existing frame image, a new subsequent associated frame image is generated through an AI artificial intelligence model tool, thereby achieving the continuity of the entire video.

[0010] Furthermore, the generation of a new subsequent associated frame image through the AI ​​artificial intelligence model tool is further as follows: a block matching algorithm is used to calculate the motion vector between adjacent frames, and according to the motion vector, pixels of the previous frame and the next frame are compensated to generate a new frame, and bilinear interpolation or bicubic interpolation method is used to further improve the quality of the new frame.

[0011] An optimization system for improving visual effects, the system comprising a monitoring module, an integrated toolkit module, and a frame supplementation module;

[0012] The monitoring module: The client first starts an asynchronous monitoring thread, which mainly monitors the computing power of the client device over a period of time, monitors the resource usage of the CPU and GPU over a period of time, and divides the remaining computing resources into three levels: high, medium, and low;

[0013] The integrated toolkit module: The client integrates a frame interpolation algorithm toolkit, which can achieve the effect of interpolating frames for some frames in the video stream, using the existing content of the previous frame to generate the content of this frame and directly replace it;

[0014] The frame-filling modules are as follows: when the remaining resources are at a high level, a method of extracting frames at a large interval is adopted, and a toolkit is used to perform frame-filling operations, and the remaining frames are still calculated and optimized according to the old method; when the remaining resources are at a medium level, a method of extracting frames at a medium interval is adopted, and a toolkit is used to perform frame-filling operations; when the remaining resources are at a low level, a method of extracting frames at a small interval is adopted, and a toolkit is used to perform frame-filling operations.

[0015] Furthermore, the monitoring module is divided into three levels: high, medium and low according to the remaining computing resources: when the remaining resources are greater than 60%, it is a high level; when the remaining resources are between 30% and 60%, it is a medium level; when the remaining resources are less than 30%, it is a low level.

[0016] Furthermore, the content of generating this frame in the integrated toolkit module is directly replaced by: based on an existing frame image, a new subsequent associated frame image is generated through the AI ​​artificial intelligence model tool, thereby achieving the continuity of the entire video.

[0017] Furthermore, the generation of a new subsequent associated frame image through the AI ​​artificial intelligence model tool is further as follows: a block matching algorithm is used to calculate the motion vector between adjacent frames, and according to the motion vector, pixels of the previous frame and the next frame are compensated to generate a new frame, and bilinear interpolation or bicubic interpolation method is used to further improve the quality of the new frame.

[0018] The beneficial effects of the present invention are as follows: for the small-scale algorithm integrated with the client, the client device types are preferentially divided into three types: high, medium and low. The high-end devices are prioritized to ensure the effect, and the low-end devices are prioritized to ensure the performance. In the small-scale algorithm, the original frame-by-frame conversion method is replaced. The high-end devices use a certain interval to add 1 frame to reduce frame conversion. The medium-end devices are similar. The low-end devices add more frames to reduce the consumption of frame conversion (adding frames will reduce resource occupation than direct frame data conversion). In this way, frame addition operations for different video special effects are performed according to the configurations of different devices, and finally the best viewing effect of the same video on different devices is achieved, thereby improving the user experience of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flow chart of the method of the present invention.

[0020] Figure 2 It is a system principle block diagram of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings.

[0022] See also Figure 1 As shown, an optimization method for improving visual effects of the present invention comprises the following steps:

[0023] Step S1, the client first starts an asynchronous monitoring thread, which mainly monitors the computing power of the client device for a period of time, monitors the resource usage of the client CPU and GPU within a period of time, and divides it into three levels of high, medium and low according to the remaining computing resources;

[0024] Step S2: The client integrates a frame interpolation algorithm toolkit, which can achieve the effect of interpolating some frames in the video stream, using the existing content of the previous frame to generate the content of this frame and directly replace it;

[0025] Step S3: When the remaining resources are of high quality, a large number of frames are extracted at intervals, and a toolkit is used to perform frame interpolation. The remaining frames are still calculated and optimized in the old way; when the remaining resources are of medium quality, a medium number of frames are extracted at intervals, and a toolkit is used to perform frame interpolation. When the remaining resources are of low quality, a small number of frames are extracted at intervals, and a toolkit is used to perform frame interpolation. The old way is to use a frame-by-frame analysis method, and each frame in the video stream is analyzed and displayed by CPU or GPU calculation.

[0026] The present invention will be further described below in conjunction with a specific embodiment:

[0027] An optimization method to improve visual effects,

[0028] 1. The client first starts an asynchronous monitoring thread, which mainly monitors the computing power of the client device over a period of time, monitors the resource usage of the client CPU and GPU over a period of time, and divides it into three levels: high, medium, and low according to the remaining computing resources;

[0029] 1.1 According to the remaining computing resources, it is divided into three levels: high, medium and low. When the remaining resources are greater than 60%, it is high level; when the remaining resources are between 30% and 60%, it is medium level; when the remaining resources are less than 30%, it is low level.

[0030] 2. The client integrates a frame interpolation algorithm toolkit, which can achieve the effect of interpolating some frames in the video stream, using the existing content of the previous frame to generate the content of this frame and directly replace it; this method only consumes computing resources, while the old method of optimizing each frame consumes multiple resources. This method of replacing and interpolating frames can reduce consumption and improve efficiency. Suppose a video stream has 100 frames of special effects. In the old method, it is necessary to optimize and calculate each of these 100 frames, and finally generate the required results to display to the client. In this solution, special processing is performed on the three grades of high, medium and low devices counted in step 1 above.

[0031] 2.1 Generate the content of this frame and directly replace it further: based on an existing frame image, generate a new subsequent associated frame image through the AI ​​artificial intelligence model tool, so as to achieve the continuity of the entire video.

[0032] 2.1.1 The method of generating a new subsequently associated frame image through the AI ​​artificial intelligence model tool is further as follows: a block matching algorithm is used to calculate the motion vector between adjacent frames, and pixels of the previous frame and the next frame are compensated according to the motion vector to generate a new frame, and a bilinear interpolation or bicubic interpolation method is used to further improve the quality of the new frame.

[0033] Input the input parameters to the AI ​​model, and the large model can output the output parameters.

[0034] For example, the frame interpolation or interpolation operation of the frame interpolation algorithm toolkit is usually to improve video smoothness, reduce blur effects or increase frame rate. These technologies are very commonly used in video editing, motion image analysis, and computer vision. 1. Frame Interpolation Method

[0035] Frame interpolation usually involves generating new frames between two frames to increase the frame rate of the video. Common interpolation methods include:

[0036] 1.1 Motion compensation based methods

[0037] The basic idea of ​​this method is to analyze the motion information between known frames and then generate intermediate frames based on this information.

[0038] step:

[0039] (1) Motion Estimation:

[0040] The motion vector between adjacent frames is calculated using methods such as the Block Matching Algorithm (BMA). The motion vector describes the movement of pixels in time.

[0041] (2) Motion Compensation:

[0042] According to the motion vector, the pixels of the previous frame and the next frame are compensated to generate a new frame.

[0043] For example, for the previous frame (I_0) and the next frame (I_1), calculate the intermediate frame (I_{0.5}) between them: [I_{0.5}(x,y)=\frac{1}{2}\left(I_0(x,y)+I_1(x+\Delta x,y+\Delta y)\right)] where (\Delta x) and (\Delta y) are motion vectors.

[0044] (3) Interpolation:

[0045] Methods such as bilinear interpolation or bicubic interpolation can be used to further improve the quality of the new frame.

[0046] 3. Example of specific algorithm formula for interpolation

[0047] Here is a simple formula example. Assuming you already have two frames of images (I_1) and (I_2), we can generate the intermediate frame by weighted averaging:

[0048] Python

[0049] import numpy as np def interpolate_frames(I1,I2,alpha):

[0050] #alpha is the interpolation factor, range [0,1]

[0051] return(1-alpha)*I1+alpha*I2

[0052] #Example Usage

[0053] I1 = np.array(...) # previous frame

[0054] I2 = np.array(...) #next frame

[0055] new_frame=interpolate_frames(I1,I2,0.5)#Generate new frames

[0056] 3. When the remaining resources are of high quality, a large number of interval frame extraction is adopted, and the toolkit is used to perform frame interpolation operations. The remaining frames are still calculated and optimized in the old way; when the remaining resources are of medium quality, a medium number of interval frame extraction is adopted, and the toolkit is used to perform frame interpolation operations; when the remaining resources are of low quality, a small number of interval frame extraction is adopted, and the toolkit is used to perform frame interpolation operations. When it is of high quality, the interval frame extraction method is adopted, for example, one frame is extracted every 8 frames, and the frame interpolation operation is performed. The remaining frames are still calculated and optimized in the old way. This method can reduce some resource consumption and improve efficiency compared with the old method of extracting all frames. When it is of medium quality, by analogy, the frame extraction and interpolation operation is performed every 6 frames. When it is of low quality, the frame extraction and interpolation operation is performed every 4 frames. In this way, the interpolation operation of different video special effects is performed according to the configuration of different devices, and finally the best viewing effect of the same video on different devices is achieved, which improves the user experience of the system.

[0057] For example: (1) If the remaining resources of a device are at a high level [about 80% remaining resources], the 1st, 9th, 17th, etc. frames of the video stream will be extracted, but instead of using the old calculation optimization, the toolkit will be used to perform frame interpolation operations to generate new frame images [frame interpolation will reduce resource usage compared to direct frame data conversion]. At this time, only about 12 frames of frame data will be operated using frame interpolation. At this time, less CPU or GPU computing resources will be consumed.

[0058] (2) When the remaining resources of the device are at a medium level, [approximately 50% of the remaining resources], the frame interpolation operation is performed at an interval of 6 frames. At this time, there will be approximately 16 frames of frame data that will be interpolated. This method will reduce some resource usage compared to the method of using optimized calculations for all 16 frames.

[0059] (2) Similarly, when the remaining resources are at a low level, [approximately 20% of the remaining resources], the interpolation is performed at intervals of 4 frames. At this time, approximately 25 frames of data will be interpolated, further reducing the consumption of the remaining resources.

[0060] See also Figure 2 As shown, the present invention also provides an optimization system for improving visual effects, the system includes a monitoring module, an integrated toolkit module, and a frame supplement module;

[0061] The monitoring module: The client first starts an asynchronous monitoring thread, which mainly monitors the computing power of the client device over a period of time, monitors the resource usage of the CPU and GPU over a period of time, and divides the remaining computing resources into three levels: high, medium, and low;

[0062] The monitoring module is divided into three levels: high, medium and low according to the remaining computing resources: when the remaining resources are greater than 60%, it is a high level; when the remaining resources are between 30% and 60%, it is a medium level; when the remaining resources are less than 30%, it is a low level.

[0063] The integrated toolkit module: The client integrates a frame interpolation algorithm toolkit, which can achieve the effect of interpolating some frames in the video stream, use the existing content of the previous frame, generate the content of this frame, and directly replace it; this method will only consume computing resources, while the old method of optimizing each frame will consume a variety of resources. This method of replacing and interpolating frames can reduce consumption and improve efficiency. Suppose a video stream has 100 frames of special effects. Under the old method, it is necessary to optimize and calculate each of the 100 frames, and finally generate the required results to display to the client. In this solution, special processing is performed on the three grades of high, medium and low devices counted in step 1 above.

[0064] The content of this frame generated in the integrated toolkit module is directly replaced by: based on an existing frame image, a new subsequent associated frame image is generated through the AI ​​artificial intelligence model tool, thereby achieving the continuity of the entire video.

[0065] The method of generating a new subsequently associated frame image through the AI ​​artificial intelligence model tool is further as follows: a block matching algorithm is used to calculate the motion vector between adjacent frames, and pixels of the previous frame and the next frame are compensated according to the motion vector to generate a new frame, and a bilinear interpolation or bicubic interpolation method is used to further improve the quality of the new frame.

[0066] Input the input parameters to the AI ​​model, and the large model can output the output parameters.

[0067] For example, the frame interpolation or interpolation operation of the frame interpolation algorithm toolkit is usually to improve video smoothness, reduce blur effects or increase frame rate. These technologies are very commonly used in video editing, motion image analysis, and computer vision. 1. Frame Interpolation Method

[0068] Frame interpolation usually involves generating new frames between two frames to increase the frame rate of the video. Common interpolation methods include:

[0069] 1.1 Motion compensation based methods

[0070] The basic idea of ​​this method is to analyze the motion information between known frames and then generate intermediate frames based on this information.

[0071] step:

[0072] (1) Motion Estimation:

[0073] The motion vector between adjacent frames is calculated using methods such as the Block Matching Algorithm (BMA). The motion vector describes the movement of pixels in time.

[0074] (2) Motion Compensation:

[0075] According to the motion vector, the pixels of the previous frame and the next frame are compensated to generate a new frame.

[0076] For example, for the previous frame (I_0) and the next frame (I_1), calculate the intermediate frame (I_{0.5}) between them: [I_{0.5}(x,y)=\frac{1}{2}\left(I_0(x,y)+I_1(x+\Delta x,y+\Delta y)\right)] where (\Delta x) and (\Delta y) are motion vectors.

[0077] (3) Interpolation:

[0078] Methods such as bilinear interpolation or bicubic interpolation can be used to further improve the quality of the new frame.

[0079] 3. Example of specific algorithm formula for interpolation

[0080] Here is a simple formula example. Assuming you already have two frames of images (I_1) and (I_2), we can generate the intermediate frame by weighted averaging:

[0081] Python

[0082] import numpy as np def interpolate_frames(I1,I2,alpha):

[0083] #alpha is the interpolation factor, range [0,1]

[0084] return(1-alpha)*I1+alpha*I2

[0085] #Example Usage

[0086] I1 = np.array(...) # previous frame

[0087] I2 = np.array(...) #next frame

[0088] new_frame=interpolate_frames(I1,I2,0.5)#Generate new frames

[0089] The frame supplementation module: when the remaining resources are of high quality, a large number of interval frame extraction is adopted, and a toolkit is used to perform frame supplementation operations, and the remaining frames are still calculated and optimized in the old way; when the remaining resources are of medium quality, a medium number of interval frame extraction is adopted, and a toolkit is used to perform frame supplementation operations; when the remaining resources are of low quality, a small number of interval frame extraction is adopted, and a toolkit is used to perform frame supplementation operations. When it is of high quality, an interval frame extraction method is adopted, for example, one frame is extracted every 8 frames, and a frame supplementation operation is performed, and the remaining frames are still calculated and optimized in the old way. This method can reduce some resource consumption and improve efficiency compared with the old method of extracting all frames. When it is of medium quality, by analogy, frame extraction and supplementation operations are performed every 6 frames. When it is of low quality, frame extraction and supplementation operations are performed every 4 frames. In this way, frame supplementation operations of different video special effects are performed according to the configuration of different devices, and finally the best viewing effect of the same video on different devices is achieved, and the user experience of the system is improved.

[0090] For example: (1) If the remaining resources of a device are at a high level [about 80% remaining resources], the 1st, 9th, 17th, etc. frames of the video stream will be extracted, but instead of using the old calculation optimization, the toolkit will be used to perform frame interpolation operations to generate new frame images [frame interpolation will reduce resource usage compared to direct frame data conversion]. At this time, only about 12 frames of frame data will be operated using frame interpolation. At this time, less CPU or GPU computing resources will be consumed.

[0091] (2) When the remaining resources of the device are at a medium level, [approximately 50% of the remaining resources], the frame interpolation operation is performed at an interval of 6 frames. At this time, there will be approximately 16 frames of frame data that will be interpolated. This method will reduce some resource usage compared to the method of using optimized calculations for all 16 frames.

[0092] (2) Similarly, when the remaining resources are at a low level, [approximately 20% of the remaining resources], the interpolation is performed at intervals of 4 frames. At this time, approximately 25 frames of data will be interpolated, further reducing the consumption of the remaining resources.

[0093] In short, the present invention is aimed at the small algorithm integrated with the client, and preferentially divides the client device types into three types: high, medium and low. The high device gives priority to ensuring the effect, and the low device gives priority to ensuring the performance. In the small algorithm, the original frame-by-frame conversion method is replaced. The high device adopts a certain interval to add 1 frame to reduce frame conversion, and the medium device is similar to it. The low device reduces the consumption of frame conversion by adding more frames (adding frames will reduce resource occupation than direct frame data conversion). In this way, frame addition operations of different video special effects are performed according to the configuration of different devices, and finally the best viewing effect of the same video on different devices is achieved, thereby improving the user experience of the system.

[0094] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. An optimization method for improving visual effects, characterized in that: The method comprises the following steps: Step S1, the client first starts an asynchronous monitoring thread, which mainly monitors the computing power of the client device for a period of time, monitors the resource usage of the client CPU and GPU within a period of time, and divides it into three levels of high, medium and low according to the remaining computing resources; Step S2: The client integrates a frame interpolation algorithm toolkit, which can achieve the effect of interpolating some frames in the video stream, using the existing content of the previous frame to generate the content of this frame and directly replace it; Step S3, when the remaining resources are of high quality, a method of extracting frames at large intervals is adopted, and a toolkit is used to perform frame supplementation operations, and the remaining frames are still calculated and optimized according to the old method; when the remaining resources are of medium quality, a method of extracting frames at medium intervals is adopted, and a toolkit is used to perform frame supplementation operations; when the remaining resources are of low quality, a method of extracting frames at small intervals is adopted, and a toolkit is used to perform frame supplementation operations.

2. The optimization method for improving visual effects according to claim 1, characterized in that: The step S1 is further divided into three levels of high, medium and low according to the remaining situation of computing resources: when the remaining resources are greater than 60%, it is a high level; when the remaining resources are between 30% and 60%, it is a medium level; when the remaining resources are less than 30%, it is a low level.

3. The optimization method for improving visual effects according to claim 1, characterized in that: The content of the frame generated in step S2 is directly replaced by: based on an existing frame image, a new subsequent associated frame image is generated through an AI artificial intelligence model tool, thereby achieving the continuity of the entire video.

4. The optimization method for improving visual effects according to claim 3, characterized in that: The method of generating a new subsequently associated frame image through the AI ​​artificial intelligence model tool is further as follows: a block matching algorithm is used to calculate the motion vector between adjacent frames, and pixels of the previous frame and the next frame are compensated according to the motion vector to generate a new frame, and a bilinear interpolation or bicubic interpolation method is used to further improve the quality of the new frame.

5. An optimization system for improving visual effects, characterized in that: The system includes a monitoring module, an integrated toolkit module, and a frame supplement module; The monitoring module: The client first starts an asynchronous monitoring thread, which mainly monitors the computing power of the client device over a period of time, monitors the resource usage of the CPU and GPU over a period of time, and divides the remaining computing resources into three levels: high, medium, and low; The integrated toolkit module: The client integrates a frame interpolation algorithm toolkit, which can achieve the effect of interpolating frames for some frames in the video stream, using the existing content of the previous frame to generate the content of this frame and directly replace it; The frame-filling modules are as follows: when the remaining resources are at a high level, a method of extracting frames at a large interval is adopted, and a toolkit is used to perform frame-filling operations, and the remaining frames are still calculated and optimized according to the old method; when the remaining resources are at a medium level, a method of extracting frames at a medium interval is adopted, and a toolkit is used to perform frame-filling operations; when the remaining resources are at a low level, a method of extracting frames at a small interval is adopted, and a toolkit is used to perform frame-filling operations.

6. The optimization system for improving visual effects according to claim 4, characterized in that: The monitoring module is divided into three levels: high, medium and low according to the remaining computing resources: when the remaining resources are greater than 60%, it is a high level; when the remaining resources are between 30% and 60%, it is a medium level; when the remaining resources are less than 30%, it is a low level.

7. The optimization system for improving visual effects according to claim 4, characterized in that: The content of this frame generated in the integrated toolkit module is directly replaced by: based on an existing frame image, a new subsequent associated frame image is generated through the AI ​​artificial intelligence model tool, thereby achieving the continuity of the entire video.

8. The optimization system for improving visual effects according to claim 7, characterized in that: The method of generating a new subsequently associated frame image through the AI ​​artificial intelligence model tool is further as follows: a block matching algorithm is used to calculate the motion vector between adjacent frames, and pixels of the previous frame and the next frame are compensated according to the motion vector to generate a new frame, and a bilinear interpolation or bicubic interpolation method is used to further improve the quality of the new frame.

Images