Video processing method and related device

Abstract

The present application provides a video processing method, comprising: obtaining a video stream to be processed, and acquiring a picture influence factor correlated with the video stream, the picture influence factor comprising a parameter that affects picture smoothness of the video stream; and in a case that the picture influence factor meets a set condition, performing motion estimation and motion compensation (MEMC) processing on the video stream, to obtain a processed video stream. Therefore, the picture influence factor-based MEMC adaptive adjustment can be implemented, so that the situation that a video picture is stuck or damaged when the MEMC is enabled can be effectively solved. In addition, the method does not need a manual operation by a user, for example, the user does not need to manually switch an MEMC parameter by using a remote control, so that both hands of the user are released, the user does not need to care about enabling or disabling of the MEMC, and the user experience is improved.

Description

Video processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 11, 2023, with application number 202311015692.2 and invention name “A video processing method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of video processing technology, and in particular to a video processing method, a video processing system, a terminal device, a computing device cluster, a computer-readable storage medium, and a computer program product. Background Art

[0003] To provide users with a better visual experience, motion estimation and motion compensation (MEMC) was developed. MEMC predicts the motion trajectory of an object and compensates between two frames, converting low-frame-rate video to high-frame-rate video, making the video smoother.

[0004] Currently, many video application scenarios are attempting to use MEMC to improve video smoothness. For example, scenarios such as TV video playback or in-person conferences support MEMC, which compensates for video distortion and enhances the viewing experience for viewers or participants. However, not all video streams are suitable for MEMC.

[0005] The industry has proposed a solution for dynamically adjusting MEMC. Taking the video playback scene on TV as an example, three motion compensation control modes are set inside the main processor of the TV, such as sports mode, game mode, and film mode. The main processor is electrically connected to the MEMC processing module, and the MEMC processing module includes MEMC parameters corresponding to the above three modes. Depending on the content of the video stream played by the TV, the user can transmit the remote control command of the corresponding mode through the remote control. The main processor of the TV receives the remote control command and instructs the MEMC processing module to apply the MEMC parameters corresponding to the mode, so that the picture quality effect (such as smoothness) in different application scenarios can reach the best state.

[0006] However, the above method relies on a pre-set motion compensation control mode and requires the user to manually switch MEMC parameters, which makes it difficult to achieve adaptive adjustment, difficult to meet business needs, and reduces user experience.

[0007] Summary of the Invention

[0008] The present application provides a video processing method that implements MEMC adaptive adjustment based on picture impact factors, which can effectively solve the problem of video freeze or damage after turning on MEMC. In addition, the method does not require manual operation by the user, and the user does not need to worry about turning MEMC on or off, thereby improving the user experience. The present application also provides a video processing system, computing device cluster, terminal device, computer-readable storage medium, and computer program product corresponding to the above method.

[0009] In the first aspect, the present application provides a video processing method. The method can be executed by a video processing system. The video processing system can be a software system, which can be deployed in a computing device cluster, such as a terminal device, or distributedly deployed in terminal devices and servers. For ease of description, the present application mainly uses the software system deployed in a terminal device for example. The terminal device executes the program code of the software system, thereby executing the video processing method of the present application. In some possible implementations, the video processing system can also be a hardware system, and the hardware system can be a computing device cluster with video processing functions, such as a terminal device with video processing functions, or a terminal device and server that collaborates to complete video processing. Among them, the terminal device can be an electronic device with MEMC function, including but not limited to computers, TVs (such as smart screen TVs), conference large screens, and smart phones. The MEMC function means that the terminal device has a MEMC module, which can insert low frame rate video into high frame rate video, and then transmit it to the display for display.

[0010] In specific implementation, the video processing system can obtain the video stream to be processed, and then collect the picture impact factor related to the video stream. The picture impact factor includes parameters that affect the picture smoothness of the video stream. When the picture impact factor meets the set conditions, the video processing system can perform motion estimation and motion compensation MEMC processing on the video stream to obtain the processed video stream.

[0011] This method adaptively adjusts MEMC based on collected image impact factors, effectively resolving video freezes or distortion after MEMC is enabled. Furthermore, this method eliminates the need for manual user interaction, such as manually switching MEMC parameters via a remote control. This frees users up to worry about turning MEMC on or off, improving the user experience.

[0012] In some possible implementations, the image influencing factors associated with the video stream include at least one of the following: the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream. This can implement an adaptive adjustment method based on the network environment and the screen resolution of the terminal device, effectively avoiding freezes and image quality degradation caused by the network environment, and avoiding the insignificant effect of turning on MEMC due to a small screen, which instead introduces additional delays.

[0013] In some possible implementations, the image impact factor associated with a video stream includes the motion state of a target object in the video stream. Accordingly, the video processing system can obtain a motion vector based on a first video frame and a second video frame in the video stream, for example, adjacent frames in the video stream. The video processing system can then perform statistical analysis on the motion vectors and classify them based on the statistical analysis results to obtain the motion state of the target object in the video stream.

[0014] In this way, an adaptive adjustment method based on motion status can be implemented. By analyzing the motion status of the target object in the video stream, it is decided whether to turn on MEMC. According to the above decision, MEMC is turned on in the appropriate time period to ensure the overall picture quality effect (such as smoothness).

[0015] In some possible implementations, the statistical analysis results of the motion vectors include at least one of the number, maximum value, mean value, and entropy value of the motion vectors. Accordingly, when the number of motion vectors is greater than a first number and less than a second number, and the mean value of the motion vectors is greater than a first threshold value and less than a second threshold value, the video processing system can determine that the motion state is a normal movement state; when the maximum value of the motion vectors is greater than a third threshold value, the video processing system can determine that the motion state is a first abnormal movement state; when the mean value of the motion vectors is less than the first threshold value, the video processing system can determine that the motion state is a stationary state; when the number of motion vectors is less than the first number, and the mean value of the motion vectors is greater than the second threshold value, the video processing system can determine that the motion state is a second abnormal movement state; and when the entropy value of the motion vectors is greater than a fourth threshold value, the video processing system can determine that the motion state is a third abnormal movement state.

[0016] This method classifies the motion state by combining the number of motion vectors, the mean of motion vectors, and the entropy of motion vectors, thereby helping to decide whether to turn on MEMC.

[0017] In some possible implementations, the picture impact factor satisfies the set conditions, for example, the resolution of the terminal device used to play the video stream is greater than the set resolution, the state parameter of the network device used to transmit the video stream is less than the set value, and / or the motion state of the target object in the video stream is a normal moving state.

[0018] In other words, the video processing system can perform MEMC processing on the video stream to obtain a processed video stream when the resolution of the terminal device used to play the video stream is greater than the set resolution; and / or, the video processing system can perform MEMC processing on the video stream to obtain a processed video stream when the state parameter of the network device used to transmit the video stream is less than a set value; and / or, the video processing system can perform MEMC processing on the video stream to obtain a processed video stream when the motion state of the target object in the video stream is a normal moving state.

[0019] In this way, when the resolution of the terminal device is greater than the set resolution, the status parameter of the network device is less than the set value and / or the motion state of the target object in the video stream is normal, MEMC can be adaptively turned on to ensure the overall picture quality when the video stream is played.

[0020] In some possible implementations, a video stream includes a processed interval, a cached interval, and subsequent video frames. The processed interval includes video frames that have undergone MEMC processing, and the cached interval is used to cache video frames in the video stream to be processed. When the MEMC state is switched, the video processing system may further perform a frame discarding operation every n frames, starting from the video frames cached in the cached interval, until the number of discarded frames meets a target value, at which point the frame discarding operation ceases.

[0021] This method achieves smooth switching of MEMC states by dropping frames evenly at intervals, avoiding black screen or freeze caused by directly clearing the cache interval (cache queue), and optimizing the video stream when switching MEMC states.

[0022] In some possible implementations, the video stream to be processed is a conference video stream captured by a conference terminal. This method collects image impact factors related to the conference video stream and adaptively adjusts the MEMC based on these image impact factors, effectively resolving issues such as conference image freezes and distortion after MEMC is enabled. This also frees the user's hands, as they no longer need to worry about turning the MEMC on and off.

[0023] In a second aspect, the present application provides a video processing system. The system comprises:

[0024] A communication module, used to obtain the video stream to be processed;

[0025] An analysis and decision module, configured to collect picture impact factors related to the video stream, wherein the picture impact factors include parameters that affect the picture smoothness of the video stream;

[0026] The motion estimation and motion compensation MEMC module is used to perform MEMC processing on the video stream when the picture impact factor meets the set conditions to obtain the processed video stream.

[0027] In some possible implementations, the picture impact factors related to the video stream include at least one of the following: the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream.

[0028] In some possible implementations, the image impact factor related to the video stream includes a motion state of a target object in the video stream, and the analysis and decision module is specifically configured to:

[0029] Obtaining a motion vector according to a first video frame and a second video frame in the video stream;

[0030] Statistical analysis is performed on the motion vectors, and classification is performed according to the statistical analysis results of the motion vectors to obtain the motion state of the target object in the video stream.

[0031] In some possible implementations, the statistical analysis result of the motion vector includes at least one of the number, maximum value, mean value, and entropy value of the motion vector, and the analysis and decision module is specifically configured to:

[0032] When the number of the motion vectors is greater than a first number and less than a second number, and the average of the motion vectors is greater than a first threshold and less than a second threshold, determining that the motion state is a normal movement state;

[0033] When the maximum value of the motion vector is greater than a third threshold, determining that the motion state is a first abnormal movement state;

[0034] When the mean value of the motion vectors is less than the first threshold, determining that the motion state is a stationary state;

[0035] When the number of the motion vectors is less than the first number and the average of the motion vectors is greater than the second threshold, determining that the motion state is a second abnormal movement state;

[0036] When the entropy value of the motion vector is greater than a fourth threshold, it is determined that the motion state is a third abnormal movement state.

[0037] In some possible implementations, the MEMC module is specifically used to:

[0038] When the resolution of the terminal device used to play the video stream is greater than the set resolution, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or,

[0039] When a state parameter of a network device used to transmit the video stream is less than a set value, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or,

[0040] When the motion state of the target object in the video stream is a normal moving state, motion estimation and motion compensation MEMC processing is performed on the video stream to obtain a processed video stream.

[0041] In some possible implementations, the video stream includes a processed interval, a cache interval, and subsequent video frames, the processed interval includes video frames that have been MEMC-processed, the cache interval is used to cache video frames to be processed in the video stream, and the system further includes:

[0042] The state switching processing module is used to, when the MEMC state is switched, take the video frame cached in the cache interval as the starting position, perform a frame dropping operation every n frames until the number of dropped frames meets the target value, and stop the frame dropping operation.

[0043] In some possible implementations, the video stream to be processed is a conference video stream collected by a conference terminal.

[0044] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, each of which includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, causing the computing device or computing device cluster to perform the video processing method described in the first aspect or any implementation of the first aspect.

[0045] In a fourth aspect, the present application provides a terminal device. The terminal device includes a processor and a memory, wherein the memory stores computer-readable instructions; the processor is configured to execute the computer-readable instructions so that the terminal device performs the video processing method described in the first aspect or any implementation of the first aspect.

[0046] In a fifth aspect, the present application provides a computer-readable storage medium having stored therein instructions that instruct a terminal device or a computing device cluster to execute the video processing method described in the first aspect or any one of the implementations of the first aspect.

[0047] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a terminal device or a computing device cluster, causes the terminal device or the computing device cluster to execute the video processing method described in the first aspect or any implementation of the first aspect.

[0048] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0050] FIG1 is a schematic diagram of the structure of a video processing system provided in an embodiment of the present application;

[0051] FIG2 is a schematic diagram of video stream optimization during MEMC state switching provided by an embodiment of the present application;

[0052] FIG3 is a flow chart of a video processing method provided in an embodiment of the present application;

[0053] FIG4 is a schematic diagram of a scenario of adaptively adjusting the application of MEMC in a resident conference based on resolution and network environment, provided by an embodiment of the present application;

[0054] FIG5 is a schematic diagram of a scenario of an application of adaptively adjusting MEMC based on motion state in an immersive conference according to an embodiment of the present application;

[0055] FIG6 is a flow chart of a method for processing video during state switching provided by an embodiment of the present application;

[0056] FIG7 is a flow chart of an application of a video processing method provided in an embodiment of the present application in an on-site conference;

[0057] FIG8 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

[0058] FIG9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0060] First, some technical terms involved in the embodiments of this application are introduced.

[0061] Video generally refers to dynamic images captured, recorded, processed, stored, transmitted, and reproduced via electrical signals. Video streaming refers to video that supports streaming data transmission. Video streams can be processed as a stable, continuous stream over the network, allowing end devices to play the video without waiting for the entire video data to be transmitted. Video streams can be pre-generated, such as pre-recorded television program streams. Video streams can also be captured in real time. For example, in a video conferencing scenario, conference endpoints can use cameras to capture images in real time to generate a video stream.

[0062] A video stream usually includes a certain number of images, which are also called frames or video frames. The picture quality (such as picture smoothness) when the video stream is played is usually related to the frame rate of the video stream. Among them, the frame rate is also called the picture update rate or refresh rate, which refers to the number of static pictures played per second. Therefore, it is also called the number of frames per second (FPS). Typical frame rates range from 6 or 8 frames per second in the early days to 120 frames per second today. In order to achieve basic visual persistence effects, the frame rate usually needs to reach 10FPS. For example, a common frame rate for movies is 24FPS. Generally speaking, the higher the frame rate, the smoother the picture when the video stream is played.

[0063] Terminal devices that use liquid crystal displays, such as LCD TVs and conference terminals, usually use electric current to stimulate liquid crystal molecules to generate points, lines, and surfaces that cooperate with the back light tubes to form the picture. Liquid crystal molecules will only work when they receive a signal, and once the picture changes too quickly, it is difficult for the liquid crystal molecules to respond in time. At this time, the terminal device still displays the picture of the previous second, which is intuitively manifested as a blurred picture. In order to improve the above-mentioned picture blur (for example, motion blur), the internal circuit processing technology can be improved to increase the number of video frames displayed per unit time. Accordingly, the video stream can also perform video processing such as video interpolation to adapt to the terminal device to improve the smoothness of the picture.

[0064] Video interpolation involves inserting new frames between adjacent frames in a video stream to increase the frame rate. This can be accomplished in a variety of ways, such as inserting black frames, inserting duplicate frames, and inserting computational frames. Motion estimation and motion compensation (MEMC) is an example of computational frame interpolation. MEMC intelligently analyzes two adjacent frames to automatically generate a new frame between the original two. This not only creates a smoother image but also effectively reduces image smearing.

[0065] However, not all video streams are suitable for MEMC. MEMC may make calculation errors or fail when facing complex scenes. For example, MEMC can estimate the motion trajectory of a small number of moving objects, but when a large number of moving objects suddenly appear in the picture, the effectiveness of MEMC will be greatly reduced.

[0066] To this end, most manufacturers have provided a switch option for the MEMC function, allowing users to freely choose according to the actual viewing screen. Some manufacturers even provide different gears or modes of MEMC, allowing users to choose the gear or mode that suits the video stream being played. For example, TV manufacturers set up a variety of motion compensation control modes inside the main processor of the TV, including but not limited to sports mode, game mode, and film and television mode. The MEMC processing module electrically connected to the main processor includes MEMC parameters corresponding to the above modes. Depending on the content of the video stream played by the TV, the user can transmit a remote control command of the corresponding mode through the remote control. The main processor of the TV receives the remote control command and instructs the MEMC processing module to apply the MEMC parameters corresponding to the mode, so that the picture quality effect (such as smoothness) in different application scenarios is optimal.

[0067] The above method relies on a pre-set motion compensation control mode and requires the user to manually switch MEMC parameters, which makes it difficult to achieve adaptive adjustment, difficult to meet business needs, and reduces user experience.

[0068] In view of this, the present application provides a video processing method. The method can be executed by a video processing system. The video processing system can be a software system, which can be deployed in a computing device cluster, such as a terminal device, or distributedly deployed in terminal devices and servers. For ease of description, the present application mainly uses the software system deployed in a terminal device for example. The terminal device executes the program code of the software system, thereby executing the video processing method of the present application. In some possible implementations, the video processing system can also be a hardware system, and the hardware system can be a computing device cluster with video processing functions, such as a terminal device with video processing functions, or a terminal device and server that collaborates to complete video processing. Among them, the terminal device can be an electronic device with MEMC function, including but not limited to computers, TVs (such as smart screen TVs), conference large screens, and smart phones. The MEMC function means that the terminal device has a MEMC module, which can insert low frame rate video into high frame rate video, and then transmit it to the display for display.

[0069] In specific implementation, the video processing system obtains the video stream to be processed and collects the picture impact factor related to the video stream. The picture impact factor includes parameters that affect the picture smoothness of the video stream. When the picture impact factor meets the set conditions, the above video stream is subjected to motion estimation and motion compensation MEMC processing to obtain the processed video stream.

[0070] This allows for adaptive MEMC adjustment based on image influencing factors, effectively resolving issues with video freezes or distortion after MEMC is enabled. Furthermore, this method eliminates the need for manual user interaction, such as manually switching MEMC parameters via a remote control. This frees users up, freeing them from worrying about turning MEMC on or off, improving user experience.

[0071] The video processing method of the present application can be applied to a variety of video processing scenarios, including but not limited to TV video playback, video live broadcast or video conferencing scenarios. Especially in the scenario of resident conference (users build a complete conference system by themselves and can enjoy conference services for life), by collecting the resolution of conference terminals (such as dedicated conference terminals such as large conference screens, or general conference terminals such as computers and mobile phones that support video conferencing), conference network conditions (such as the status parameters of network devices used to transmit conference video streams), the motion status of target objects in conference video streams and other picture influencing factors, and automatically starting MEMC processing of the video stream when the picture influencing factors meet the conditions, thereby solving the problems of picture freeze and picture quality damage caused by poor network status, fast movement of participants in the scene or frequent small-area movements in the scene.

[0072] In order to make the technical solution of this application clearer and easier to understand, the architecture of the video processing system of this application is introduced below using an immersive conference scenario.

[0073] Referring to the structural diagram of a video processing system shown in FIG1 , the video processing system can be deployed in a conference terminal, for example, in the conference terminal A shown in FIG1 , and in the conference terminal B shown in FIG1 . Conference terminal A and conference terminal B can be dedicated conference terminals, such as a large conference screen, or general-purpose conference terminals, such as a computer or a mobile phone. When conference terminal A and conference terminal B participate in the same conference, conference terminal B can execute the program code of the video processing system to process the conference video stream sent by conference terminal A. It should be noted that conference terminal A can also execute the program code of the video processing system to process the conference video stream sent by conference terminal B. This embodiment uses the processing of the conference video stream sent by conference terminal A by conference terminal B as an example.

[0074] Specifically, conference terminal B is deployed with a video processing system 100, which includes a communication module (not shown in Figure 1), an analysis and decision module 102, and a MEMC module 104. The communication module is used to obtain a video stream to be processed, which may be a conference video stream, such as a conference video stream generated by conference terminal A in real time. The analysis and decision module 102 is used to collect image impact factors related to the video stream, which include parameters that affect the image smoothness of the video stream. The image impact factors related to the video stream include at least one of the following: the resolution of the terminal device used to play the video stream (for example, conference terminal B), the status parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream. In some examples, the analysis and decision module 102 may include one or more of a resolution and network status analysis and decision module 102A and a motion status analysis and decision module 102B. The MEMC module 104 is used to perform MEMC processing on the video stream when the image impact factor meets the set conditions to obtain the processed video stream. In this way, MEMC can be adaptively adjusted based on resolution or network status, or based on the motion status of objects in the scene. The video stream processed by MEMC module 104 can be transmitted to a display for display. Compared with directly displaying the original video stream, displaying the processed video stream can achieve better fluency and provide better visual effects.

[0075] It should be noted that whether the image impact factor meets the set conditions can be determined by the analysis and decision module 102, and a corresponding MEMC decision can be generated. The MEMC decision is used to indicate whether to perform MEMC processing (or whether to enable MEMC).

[0076] Furthermore, MEMC state switching can be generated during the adaptive adjustment of MEMC. For example, as the meeting progresses, the motion state of the participants or other objects appearing in the meeting screen (such as a clock) may change, and accordingly, the MEMC state may also change. The following is explained with reference to examples. In this example, when the participants (for example, the participants in the conference video stream collected by conference terminal A) are walking normally, MEMC can be turned on; when the participants continue to sit in a fixed position and are in a stationary state, MEMC can be turned off. When a MEMC state switch occurs, for example, when the MEMC state switches from an on state to an off state, a momentary black screen or freeze may occur. This application also supports processing the video stream to solve the black screen or freeze problem caused by the MEMC state switch.

[0077] In a specific implementation, the video processing system 100 may further include a state switching processing module 106. After MEMC is turned on, the video stream may include a processed interval, a cache interval and subsequent video frames. As shown in Figure 2, the processed interval is also referred to as an inserted frame display interval, and the processed interval includes video frames that have been MEMC processed. The cache interval may include video frames to be MEMC processed. The state switching processing module 106 is used to, when the MEMC state is switched, use the video frames cached in the cache interval as the starting position, and perform a frame discarding operation every n frames until the number of dropped frames meets the target value, and the frame discarding operation is stopped. For example, the state switching processing module 106 can count the length of the cache interval, starting from the first frame of the cache queue, and discard one frame every N frames until the number of dropped frames reaches the length of the cache interval, and the frame discarding is stopped. In this way, a smooth transition of the MEMC state can be completed, avoiding the generation of instantaneous black screen or freeze, which affects the visual effect.

[0078] Based on the video processing system 100 shown in Figure 1, the present application further provides a video processing method. The video processing method of the present application is described below in conjunction with an embodiment.

[0079] Referring to the flowchart of a video processing method shown in FIG3 , the method includes:

[0080] S302: The video processing system 100 obtains a video stream to be processed.

[0081] The video stream to be processed may be a video stream for which a decision is to be made whether to perform MEMC processing to improve the degree of picture flow. The video stream to be processed may be a real-time recorded video stream, or a pre-generated video stream. For example, when the video processing system 100 is a video processing system deployed in a terminal device such as a television, the video stream to be processed may be a video stream pushed by the server. Depending on the signal source, the video stream pushed by the server may be a pre-generated video stream (such as a video stream of a pre-produced TV series or movie), or a real-time recorded video stream (such as a live stream collected during a live TV broadcast). For another example, when the video processing system 100 is a video processing system deployed in a conference terminal, the video stream to be processed may be a video stream recorded in real time by the conference terminals of other participants.

[0082] S304: The video processing system 100 collects the image impact factor related to the video stream. When the image impact factor meets the set conditions, S306 is executed.

[0083] The picture impact factor includes parameters that affect the smoothness of the video stream. In some possible implementations, the picture impact factor includes the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream (also known as the network state), and the motion state of the target object in the video stream.

[0084] Among them, when the resolution of the terminal device is small (the screen is small), the effect of MEMC is usually not very significant. Instead, it will introduce additional delays (such as the delay of MEMC processing), which in turn affects the smoothness of the picture. If the network environment is poor and the network status is not good, turning on MEMC can cause the jamming to be more obvious. In addition, if the image quality of the video stream is compressed due to poor network status, it will also affect the smoothness of the picture. Even if MEMC is turned on, the improvement in the smoothness of the picture is relatively limited. Moreover, the motion state of the target object in the video stream is different, and the degree of improvement in the visual effect of turning on MEMC can also be different. For example, in some motion states, the visual effect (degree of picture flow) of turning on MEMC is relatively limited, and in other states, the visual effect (picture smoothness) of turning on MEMC is more significantly improved. Therefore, the video processing system 100 can collect at least one of the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream to assist in deciding whether to turn on MEMC.

[0085] In a specific implementation, the video processing system 100 can determine whether the image impact factor meets the set conditions, thereby determining whether to turn on MEMC. When the image impact factor meets the set conditions, MEMC can be turned on to execute S306, and then the processed video stream is transmitted to the display for display to improve the visual effect. When the image impact factor does not meet the set conditions, MEMC processing is not performed, and the video processing system 100 can transmit the original video stream to the display for display.

[0086] The setting conditions may be set according to business needs. In some examples, the setting conditions may include at least one of the following:

[0087] The resolution of the terminal device used to play the video stream is greater than the set resolution;

[0088] The status parameter of the network device used to transmit the video stream is less than the set value;

[0089] The motion scene of the target object in the video stream is a normal moving state.

[0090] The normal movement state refers to an object being in motion and its movement speed being within a set range. In specific implementations, the video processing system 100 can determine whether an object is in a normal movement state based on motion vectors. In some examples, if the number and mean of motion vectors are within a certain range, for example, if the number of motion vectors is greater than a first number and less than a second number, and the mean of motion vectors is greater than a first threshold and less than a second threshold, then the object can be determined to be in a normal movement state.

[0091] S306 : The video processing system 100 performs MEMC processing on the video stream to obtain a processed video stream.

[0092] Specifically, the video processing system 100 can perform MEMC processing on the video stream frame by frame to obtain a processed video stream. During the MEMC processing, the video stream can be divided into a processed interval, a cache interval, and subsequent video frames. The processed interval includes video frames that have been MEMC processed, and the cache interval is used to cache video frames to be processed in the video stream. The video frames in the processed interval can be transmitted to the display for display in a time sequence. Accordingly, the video processing system 100 can obtain video frames from the cache interval for MEMC processing, and then add the video frames that have been MEMC processed to the processed interval. In addition, the video processing system 100 can obtain video frames from subsequent video frames and add them to the cache interval to continue MEMC processing.

[0093] Based on the above description, it can be seen that the video processing method provided by the embodiment of the present application can realize MEMC adaptive adjustment based on the picture influencing factor, which can effectively solve the problem of video freezing or damage after turning on MEMC. In addition, this method does not require manual operation by the user, for example, it does not require the user to manually switch MEMC parameters through the remote control, freeing the user's hands and the user does not need to worry about turning MEMC on or off, thereby improving the user experience.

[0094] The video processing method of the embodiment of the present application is introduced in detail below using a conference scenario.

[0095] Taking into account the requirements of process and real-time performance, conference scenes usually cannot frequently experience freezes and high delays. To this end, the video processing system 100 can adaptively adjust MEMC based on resolution and network environment.

[0096] Among them, the processing logic of the video processing system 100 for adaptively adjusting MEMC based on resolution and network environment can be: obtaining the resolution (denoted as curScreenResolution) of the current conference terminal (such as conference terminal B). If the resolution of the current conference terminal is lower than the minimum resolution required by MEMC (denoted as minScreenResolution), since turning on MEMC will only improve the visual perception of smoothness to a limited extent and will introduce additional delays, in other words, the disadvantages of turning on MEMC at this time outweigh the benefits. Therefore, if the resolution is lower than the above-mentioned minimum resolution (set resolution), the video processing system 100 can turn off MEMC.

[0097] Furthermore, the video processing system 100 can also obtain status parameters of the network device used to transmit the video stream, including but not limited to the current network packet loss rate curPacketLossRate, latency curLatency, and jitter curDithering, wherein each status parameter is set with a corresponding threshold value, which is usually a set value, for example, it can be set according to the bandwidth (for example, the bandwidth allocated to the meeting in the conference scenario). In some examples, the threshold values ​​of packet loss rate packetLossRateThreshold, latencyThreshold, and jitter ditheringThreshold corresponding to a 2M bandwidth conference are 20%, 150ms, and 50ms, respectively. The video processing system 100 can determine whether the current network's packet loss rate curPacketLossRate, latency curLatency, and jitter curDithering are less than the corresponding thresholds (set values). If not, it means that the network environment does not meet the set conditions. If MEMC is turned on, the freeze will be more obvious. Moreover, if the network environment is not good and the image quality of the video stream is compressed, it will be difficult to improve the smoothness of the picture even if MEMC is turned on. The video processing system 100 can decide to turn off MEMC, that is, when the network environment does not meet the set conditions, MEMC processing will not be performed on the video stream.

[0098] FIG4 shows a schematic diagram of a scenario for adaptively adjusting the application of MEMC in a resident conference based on resolution and network environment, which specifically includes the following steps:

[0099] S402: Conference terminal B obtains the resolution curScreenResolution of the current terminal device. If the resolution meets the set conditions, S404 is executed; if the resolution does not meet the set conditions, S408 is executed to output the original conference video stream.

[0100] The resolution meeting the set conditions may be that the resolution is greater than the set resolution (for example, the set minimum resolution). In this case, conference terminal B can continue to analyze whether other screen influencing factors meet the set conditions, and then determine whether to enable MEMC. The resolution not meeting the set conditions may be that the resolution is less than the set resolution (such as the above-mentioned minimum resolution). If MEMC is enabled, the improvement in screen flow is not significant, but rather increases latency. Therefore, S408 is executed to disable MEMC and output the original conference video stream.

[0101] S404: Conference terminal B obtains the status parameters of the current network device. If the status parameters of the current network device meet the set conditions, S406 is executed to output the processed high frame rate conference video stream. If the status parameters of the current network device do not meet the set conditions, S408 is executed to output the original conference video stream.

[0102] The state parameters that characterize the network state include at least one of the packet loss rate curPacketLossRate, the latency curLatency, and the jitter curDithering. The state parameters of the current network device that meet the set conditions may be that the packet loss rate is less than the corresponding threshold, the latency is less than the corresponding threshold, and / or the jitter is less than the corresponding threshold. Among them, the threshold corresponding to the packet loss rate, the threshold corresponding to the latency, and the threshold corresponding to the jitter can be determined according to the bandwidth (curBandwidth) of the current conference. curBandwidth corresponds to a set of thresholds (P0, L0, D0) of the state parameters that characterize the network state. If the packet loss rate curPacketLossRate is less than P0, the latency curLatency is less than L0, and the jitter curDithering is less than D0, it means that the state parameters of the current network device meet the set conditions, and the conference terminal B can execute S406 to turn on MEMC, thereby outputting a high frame rate conference video stream. If the packet loss rate curPacketLossRate is greater than P0, the delay curLatency is greater than L0, or the jitter curDithering is greater than D0, it means that the status parameters of the current network device do not meet the set conditions and the network environment is poor. Conference terminal B can execute S408 to shut down MEMC and output the original conference video stream.

[0103] S406: Conference terminal B turns on MEMC.

[0104] S408: Conference terminal B closes MEMC.

[0105] Whether it is a high-frame rate conference video stream with MEMC output turned on or an original conference video stream with MEMC output turned off, it can be transmitted to the monitor for display.

[0106] This method can, based on the parameters of the resident conference equipment (such as the conference terminal) (including but not limited to the screen resolution) and the conference network parameters (including packet loss rate, delay, and jitter) obtained in real time, promptly turn on MEMC in scenarios where MEMC can effectively improve the visual effect, and promptly turn off MEMC in scenarios where the effect is not obvious, delays are introduced, and freezes are likely to occur, thereby realizing adaptive adjustment of MEMC based on resolution and network status.

[0107] Considering that some scenes in the conference are not suitable for turning on MEMC or the visual effect of turning on MEMC is limited, the conference scene can also be analyzed in real time and MEMC can be adjusted in time according to the current scene.

[0108] In a specific implementation, the video processing system 100 deployed in the conference terminal can obtain a motion vector based on a first video frame and a second video frame in a video stream, for example, adjacent frames in the video stream. Motion vectors can effectively represent the direction and magnitude of motion between the two frames, so further analysis of the motion vectors can reveal the motion state. Specifically, the video processing system 100 can perform statistical analysis on the motion vectors and classify them based on the statistical analysis results to determine the motion state of the target object in the video stream.

[0109] The statistical analysis results of the motion vectors may include at least one of the number, maximum value, mean value, and entropy value of the motion vectors. When the number of motion vectors is greater than a first number and less than a second number, and the mean value of the motion vectors is greater than a first threshold value and less than a second threshold value, the video processing system 100 may determine that the motion state is a normal motion state. When the maximum value of the motion vectors is greater than a third threshold value, the video processing system 100 may determine that the motion state is a first abnormal motion state, which may be a fast motion state. When the number of motion vectors is less than the first number, and the mean value of the motion vectors is greater than the second threshold value, the video processing system 100 may determine that the motion state is a second abnormal motion state, which may be a frequent small-area motion state (such as a clock hand moving or a participant waving). When the entropy value of the motion vectors is greater than a fourth threshold value, the video processing system 100 may determine that the motion state is a third abnormal motion state, which may be an irregular motion state, such as a jumping state. When the mean value of the motion vectors is less than the first threshold value, the video processing system 100 may determine that the motion state is a stationary state (such as a participant sitting in a fixed position continuously without moving around).

[0110] The video processing system 100 can estimate the motion vector by optical flow method, full search method, pyramid search method, gradient method or block matching method. For ease of description, the following example illustrates the video processing system 100 estimating the motion vector by optical flow method.

[0111] The working principle of optical flow method is based on the following assumptions:

[0112] (1) The pixel intensity within a scene remains essentially unchanged between adjacent frames.

[0113] (2) Adjacent pixels have similar motion.

[0114] The optical flow estimation process is as follows:

[0115] Take a point P, at time t0, after Δdt, point P will move to P', and P' itself and its surroundings have similar brightness values ​​to P. For point P, assuming the coordinates are (x, y), then: I(x, y, t) = I(x+dx, y+dy, t+dt)

[0116] Here, I(x,y,t) represents the brightness of point P at time t. After time dt, point P has moved a distance dx, dy. Assuming the movement is small, Taylor's formula can be used to express it as:

[0117] HOT is a high-order infinitesimal, which can be obtained from the first assumption and Taylor expansion:

[0118] Then divide by Δt to get: x u+f y v+f t =0

[0119] in,

[0120] The above formula becomes the optical flow equation, where u and v represent the moving speed in two directions, and f x 、f y 、f t represents the gradient of brightness on the three axes. u,v are the calculated optical flow. It is usually difficult to use one equation to solve two unknowns, so the Lucas-Kanade (abbreviated as LK) method can be used to solve this problem:

[0121] Based on the second assumption, all adjacent pixels have the same movement. The LK method uses a 3*3 window, and the 9 pixels in the window satisfy: x u+f y v+f t =0

[0122] Substituting the points into the equation, the problem becomes solving two unknowns using 9 points. Since the equation is an overdetermined equation, the least squares method can be used to find the optimal solution. The following is the calculated result:

[0123] After obtaining the motion vector, the video processing system 100 can count the number of motion vectors motionVectorNum, the maximum value of the motion vector maxMotionVector, and the mean value of the motion vector Or the entropy of the motion vector mv When calculating the mean value, the video processing system 100 may also select the motion vectors with the top K vector lengths to calculate the mean value. When calculating the entropy value, the video processing system 100 may also select the motion vectors with the top K vector directions to calculate the entropy value. The specific calculation process is as follows: maxMotionVector = max(motionVector) entropy mv=-∑P(motionVector)logP(motionVector)

[0124] Here, k is an integer greater than 1.

[0125] The video processing system 100 can perform classification based on the statistical analysis results, obtain the motion state of the target object in the video stream, and make a decision on whether to enable MEMC based on the motion state.

[0126] When the number and mean of motion vectors are within the set range, as shown below: motionVectorNum∈(num1,num2)

[0127] Where num1 represents the first number, num2 represents the second number, and threMV1 and threMV2 represent the first threshold and the second threshold, respectively. Based on this, the video processing system 100 can determine that the participant is moving normally, indicating that the motion state of the target object is a normal moving state. Accordingly, the video processing system 100 can enable MEMC.

[0128] When the maximum value of the motion vector is greater than the third threshold, as shown below: maxMotionVector>threMV3

[0129] Wherein, threMV3 represents a third threshold value. The video processing system 100 may determine that the participant moves quickly, indicating that the motion state of the target object is a first abnormal motion state, and accordingly, the video processing system 100 may disable MEMC.

[0130] When the mean value of the motion vector is (continuously) less than the first threshold, as shown below:

[0131] This means that the participants continue to sit in fixed positions and no one moves around. The video processing system 100 can determine that the motion state of the moving object is a stationary state. At this time, the MEMC gain is limited, which introduces additional delay. The video processing system 100 can turn off MEMC.

[0132] When the number of motion vectors is less than the first number, the mean of the motion vectors is greater than the second threshold, as shown below:

[0133] motionVectorNum <num1

[0134] It indicates that there are frequent small-area movements in the scene (the number of motion vectors is less than a certain threshold and the mean is greater than a certain threshold), such as clock hands, participants waving hands, etc. The video processing system 100 can determine that the motion state of the target object is the second abnormal movement state. Accordingly, the video processing system 100 can turn off MEMC.

[0135] When the entropy value of the motion vector is (continuously) greater than the fourth threshold, as shown below: mv >threE

[0136] Among them, entropy mv represents the entropy value of the motion vector, threE represents the fourth threshold, and if the entropy value is continuously greater than the fourth threshold, it indicates that there is irregular motion in the scene, such as jumping. The video processing system 100 can determine that the motion state of the target object is the third abnormal movement state. Accordingly, the video processing system 100 can turn off MEMC.

[0137] FIG5 shows a schematic diagram of a scenario for adaptively adjusting the application of MEMC in a resident conference based on motion state, which specifically includes the following steps:

[0138] S502: Conference terminal B determines the motion vector of the target object in the original conference video stream based on adjacent frames in the video stream, and then performs statistical analysis on the motion vector to obtain a statistical analysis result of the motion vector.

[0139] Specifically, conference terminal B (for example, the video processing system 100 deployed in conference terminal B) can estimate the motion vectors of the two frames before and after the original conference video stream by the optical flow method, and count the number of motion vectors motionVectorNum, the maximum value of the motion vector maxMotionVector, and the mean value of the Top K motion vectors And the entropy of the TopK motion vector mv .

[0140] S504: Conference terminal B classifies the target object in the video stream based on the statistical analysis results of the motion vectors and obtains the motion state of the target object in the video stream. If the motion state meets the set conditions, execute S506; if the motion state does not meet the set conditions, execute S508.

[0141] The motion state meeting the set conditions may mean that the motion state is a normal motion state. The motion state not meeting the set conditions may mean that the motion state is not a normal motion state, for example, the motion state is a stationary state or an abnormal motion state. The abnormal motion state includes but is not limited to the first abnormal motion state, the second abnormal motion state, and the third abnormal motion state.

[0142] The specific implementation of classifying and determining the motion state according to the statistical analysis results of the motion vectors can be found in the relevant description above and will not be repeated here.

[0143] S506: Conference terminal B turns on MEMC.

[0144] S508: Conference terminal B turns off MEMC.

[0145] Whether it is a high-frame rate conference video stream with MEMC output turned on or an original conference video stream with MEMC output turned off, it can be transmitted to the monitor for display.

[0146] This method can use information such as motion status to classify scenes, adjust the MEMC strategy in time according to the scene category (such as motion status category), and effectively reduce the negative impact of MEMC in meeting scenes.

[0147] Furthermore, MEMC state switching can also occur during the MEMC processing, for example, switching from the MEMC on state to the MEMC off state. Directly clearing the cache queue (the queue formed by the frames in the cache interval) when switching the MEMC state may cause the conference screen to go black or freeze. For this reason, this application proposes a method of evenly dropping frames at intervals to achieve smooth switching of MEMC states and avoid black screens.

[0148] In specific implementation, when the state switching processing module 106 of the video processing system 100 detects that the MEMC state flag is changed to off, it determines the cache interval length cacheFrameNum. As shown in Figure 3, the video processing system 100 can perform frame dropping operations every few frames (for example, 1 frame) starting from the first frame of the cache interval until the number of dropped frames dropFrameNum is equal to the cache interval length cacheFrameNum, that is, the smooth switching of the MEMC state is completed. This switching method effectively solves the black screen or freeze caused by MEMC state switching.

[0149] FIG6 shows a flow chart of a method for processing video during state switching, which specifically includes the following steps:

[0150] S602: Conference terminal B checks the MEMC status. If the MEMC status has been switched, execute S604; if the MEMC status has not been switched, execute S606.

[0151] Among them, the MEMC state switching includes switching from the MEMC on state to the MEMC off state. In specific implementation, the conference terminal B is deployed with a video processing system 100. The MEMC module 104 in the video processing system 100 can maintain the MEMC state flag. When the MEMC state changes, the MEMC module 104 can modify the state flag. Accordingly, the state switching processing module 106 can read the state flag to obtain the MEMC state. When the result of the state flag read by the state switching processing module 106 this time is off, and the result of the state flag read last time is on, it indicates that the MEMC state has switched.

[0152] S604: Conference terminal B performs frame dropping processing on the conference video stream at intervals and uniformly.

[0153] Among them, conference terminal B pre-sets the frame interval for uniform frame loss, recorded as intervalFrameNum. In other words, when the interval uniform frame loss process is triggered, one frame can be dropped every intervalFrameNum frames. When conference terminal B detects that the MEMC state has switched from on to off, it can count the current cache interval length cacheFrameNum, and start from the first frame in the cache interval, and evenly drop frames every intervalFrameNum frames. Every time a frame is dropped, the frame loss count is increased by dropFrameNum+1 until the frame loss stops when dropFrameNum=cacheFrameNum. In this way, a smooth transition of the MEMC state is completed.

[0154] S606: Conference terminal B continues to perform MEMC processing on the conference video stream.

[0155] It should be noted that both the conference video stream output after the uniform frame loss processing and the conference video stream output after the MEMC processing can be transmitted to the display for display.

[0156] In order to address the problem of black screen or freeze every time the MEMC state is switched, which seriously affects the visual experience of the participants, this embodiment realizes a smooth transition of the MEMC state, which on the one hand improves the fluency of the conference screen, and on the other hand can also improve the usability of the MEMC adaptive adjustment method.

[0157] Based on the above embodiment, the present application further provides a video processing method. Referring to FIG7 , the method includes the following steps:

[0158] S702: The video processing system 100 receives a conference video stream.

[0159] S704: The video processing system 100 checks whether the MEMC state has switched. If so, the process goes to S706; if not, the process goes to S708.

[0160] The video processing system 100 may read the status flag to determine whether the MEMC state has switched, or the video processing system 100 may detect a state change event to determine whether the MEMC state has switched.

[0161] S706 : The video processing system 100 performs interval uniform frame dropping processing on the conference video stream.

[0162] The specific implementation of the video processing system 100 performing the interval uniform frame drop processing on the conference video stream can be found in the relevant description above. After the video processing system 100 performs the interval uniform frame drop processing on the conference video stream, such as a high frame rate conference video stream, it continues to execute S708 to determine whether to perform MEMC processing on subsequent video frames.

[0163] S708: The video processing system 100 obtains the resolution of the conference terminal used to play the video stream.

[0164] S710: The video processing system 100 determines whether the resolution meets the set condition. If the resolution meets the set condition, the video processing system executes S712; if the resolution does not meet the set condition, the video processing system 100 executes S722.

[0165] S712: The video processing system 100 obtains status parameters of a network device used to transmit a conference video stream.

[0166] S714: The video processing system 100 determines whether the state parameters of the network device transmitting the conference video stream meet the set conditions. If so, execute S716; if not, execute S722.

[0167] S716 : The video processing system 100 determines the motion state of the target object in the video stream according to adjacent video frames in the conference video stream.

[0168] S718: The video processing system 100 determines whether the motion state satisfies a set condition. If so, the process goes to S720; if not, the process goes to S722.

[0169] S720: The video processing system 100 turns on MEMC.

[0170] In a specific implementation, the video processing system 100 enables MEMC and performs MEMC processing on the conference video stream to output a high-frame-rate conference video stream. For example, if the original video conference stream is a 30-frame video conference stream, the video processing system 100 can output a 60-frame video conference stream after enabling MEMC.

[0171] S722: The video processing system 100 turns off the MEMC.

[0172] In a specific implementation, the video processing system 100 can output the original conference video stream by disabling MEMC. It should be noted that if the MEMC state switches from MEMC-enabled to MEMC-off, S724 can be executed. Upon detecting this state switch, evenly spaced frame drop processing can be performed and the original conference video stream can be continued after output. For example, if the original video conference stream is a 30-frame video conference stream, the video processing system 100 can output a 30-frame video conference stream after disabling MEMC.

[0173] S724: The video processing system 100 updates the status flag.

[0174] Whether MEMC is turned on or off, the video processing system 100 can update the status flag to determine whether a MEMC state switch occurs based on the status flag, thereby determining the processing of subsequent video frames. When MEMC is turned on, the video processing system 100 can update the status flag to on, and when MEMC is turned off, the video processing system 100 can update the status flag to off.

[0175] It should also be noted that the above S708 to S718 are a specific implementation of the video processing system 100 obtaining the picture influence factors related to the conference video stream and determining whether the picture influence factors meet the set conditions. In other possible implementation methods of the embodiment of the present application, the video processing system 100 can also obtain the picture influence factors according to other time sequences. For example, the picture influence factors can be obtained in parallel to determine whether each picture influence factor meets its corresponding set conditions, or the video processing system 100 can also obtain other picture influence factors and decide whether to turn on MEMC based on whether other picture influence factors meet the corresponding set conditions.

[0176] The present application further provides a video processing system 100, as shown in FIG1 , comprising:

[0177] The communication module (not shown in FIG1 ) is used to obtain the video stream to be processed.

[0178] An analysis and decision module 102 is configured to collect picture impact factors related to the video stream, wherein the picture impact factors include parameters that affect the picture smoothness of the video stream;

[0179] The MEMC module 104 is configured to perform MEMC processing on the video stream to obtain a processed video stream when the picture impact factor meets a set condition.

[0180] Exemplarily, the above-mentioned communication module, analysis and decision module 102, and MEMC module 104 may be implemented by hardware or by software.

[0181] When implemented via software, the communication module, analysis and decision module 102, and MEMC module 104 can be applications running on a computing device, such as a computing engine. These applications can be provided to users as virtualized services. Virtualized services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. VM services can use virtualization technology to create a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) to provide users with VMs on demand. BMS services use virtualized BMS resource pools on multiple physical hosts to provide users with BMSs on demand. Container services use virtualized container resource pools on multiple physical hosts to provide users with containers on demand. A VM is a simulated virtual computer, or logically a single computer. BMS is a scalable, high-performance computing service with computing performance comparable to traditional physical machines and secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are merely specific examples. In actual applications, the virtualization service may also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0182] When implemented through hardware, the communication module may be a transceiver or a transceiver unit, and the analysis and decision module 102 and the MEMC module 104 may include at least one computing device, such as a server. Alternatively, the analysis and decision module 102 and the MEMC module 104 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0183] In some possible implementations, the image impact factor associated with the video stream includes at least one of the following: the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream. The analysis and decision module 102 may include a resolution and network state analysis and decision module 102A, which is specifically used to collect the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream.

[0184] In some possible implementations, the image impact factor associated with the video stream includes the motion state of a target object in the video stream. The analysis and decision module 102 may include a motion state analysis and decision module 102B. The motion state analysis and decision module 102B is specifically configured to: obtain a motion vector based on the first and second video frames in the video stream; perform statistical analysis on the motion vector; and classify the motion vector based on the statistical analysis results to obtain the motion state of the target object in the video stream.

[0185] In some possible implementations, the statistical analysis results of the motion vectors include at least one of the number, maximum value, mean value, and entropy value of the motion vectors. The analysis and decision module 102 (for example, the motion state analysis and decision module 102B) is specifically configured to: determine that the motion state is a normal movement state when the number of the motion vectors is greater than a first number and less than a second number, and the mean value of the motion vectors is greater than a first threshold value and less than a second threshold value; determine that the motion state is a first abnormal movement state when the maximum value of the motion vectors is greater than a third threshold value; determine that the motion state is a stationary state when the mean value of the motion vectors is less than the first threshold value; determine that the motion state is a second abnormal movement state when the number of the motion vectors is less than the first number and the mean value of the motion vectors is greater than the second threshold value; and determine that the motion state is a third abnormal movement state when the entropy value of the motion vectors is greater than a fourth threshold value.

[0186] In some possible implementations, the MEMC module 104 is specifically configured to:

[0187] When the resolution of the terminal device used to play the video stream is greater than the set resolution, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or,

[0188] When a state parameter of a network device used to transmit the video stream is less than a set value, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or,

[0189] When the motion state of the target object in the video stream is a normal moving state, motion estimation and motion compensation MEMC processing is performed on the video stream to obtain a processed video stream.

[0190] In some possible implementations, the video stream includes a processed interval, a cache interval, and subsequent video frames, the processed interval includes video frames that have been MEMC-processed, the cache interval is used to cache video frames to be processed in the video stream, and the system 100 further includes:

[0191] The state switching processing module 106 is used to, when the MEMC state is switched, start from the video frame cached in the cache interval, perform a frame dropping operation every n frames until the number of dropped frames meets the target value, and then stop the frame dropping operation.

[0192] Similar to the analysis and decision module 102 and the MEMC module 104 , the state switching processing module 106 may be implemented by hardware or software.

[0193] When implemented by software, the state switching processing module 106 may be an application running on a computing device, such as a computing engine. The application may be provided to users in the form of a virtualized service. The virtualized service may include a VM service, a BMS service, and a container service.

[0194] When implemented in hardware, the state switching processing module 106 may include at least one computing device, such as a server. Alternatively, the state switching processing module 106 may be implemented using an ASIC or a PLD. The PLD may be a CPLD, an FPGA, a GAL, or any combination thereof.

[0195] In some possible implementations, the video stream to be processed is a conference video stream collected by a conference terminal.

[0196] This application also provides a terminal device 800. As shown in Figure 8, terminal device 800 includes a bus 802, a processor 804, a memory 806, a display 807, and a communication interface 808. Processor 804, memory 806, and communication interface 808 communicate with each other via bus 802. It should be understood that this application does not limit the number of processors and memories in terminal device 800.

[0197] Bus 802 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 802 may include a path for transmitting information between various components of terminal device 800 (e.g., memory 806, processor 804, and communication interface 808).

[0198] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). In some examples, the terminal device 800 may include a general-purpose CPU, and the video stream is subjected to MEMC processing by the general-purpose CPU. In other examples, the terminal device 800 may include a dedicated MEMC processor (or MEMC chip, MEMC processing module), and the general-purpose CPU and the dedicated MEMC processor collaborate to complete the MEMC processing of the video stream.

[0199] The memory 806 may include a volatile memory, such as a random access memory (RAM). The memory 806 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 806 stores executable program code, and the processor 804 executes the executable program code to implement the aforementioned video processing method. Specifically, the memory 806 stores instructions for the video processing system 100 to execute the video processing method.

[0200] Display 807 is an output device for displaying images and colors. Common displays can be computer or television screens. Display 807 can be of different types depending on the display principle. For example, display 807 can be a liquid crystal display. Display 807 can receive a video stream processed by processor 804 (such as a general-purpose CPU or a dedicated MEMC processor) and display the processed video stream. Alternatively, display 807 can receive the original video stream provided by processor 804 and display the original video stream.

[0201] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the terminal device 800 and other devices or a communication network.

[0202] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device, such as a terminal device, and at least one server. The server can be a central server, an edge server, or a local server in a local data center.

[0203] As shown in Figure 9, the computing device cluster includes the terminal device 800 shown in Figure 8 and at least one server 900. Server 900 may include a bus 902, a processor 904, a memory 906, and a communication interface 908. Processor 904, memory 906, and communication interface 908 communicate with each other via bus 902. It should be understood that this application does not limit the number of processors and memories in server 900.

[0204] The memory of the terminal device 800 and the server 900 in the computing device cluster may store the same video processing system 100 for executing instructions of the video processing method.

[0205] In some possible implementations, the terminal device 800 and the server 900 in the computing device cluster can also be used to execute some of the instructions for executing the video processing method by the video processing system 100. In other words, the combination of the terminal device 800 and the server 900 can jointly execute the instructions for executing the video processing method by the video processing system 100.

[0206] It should be noted that the memories in different devices in the computing device cluster may store different instructions for executing partial functions of the video processing system 100 .

[0207] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a terminal device or a computing device cluster to execute the above-mentioned method for executing the video processing system 100.

[0208] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device cluster or a terminal device, or stored on any available medium. When the computer program product is run on the computing device cluster or the terminal device, the computing device cluster or the terminal device executes the above-described video processing method.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A video processing method, characterized in that: The method comprises: Get the video stream to be processed; Collecting picture impact factors related to the video stream, wherein the picture impact factors include parameters that affect the picture smoothness of the video stream; When the picture impact factor meets the set condition, the video stream is subjected to motion estimation and motion compensation MEMC processing to obtain a processed video stream.

2. The method according to claim 1, characterized in that The picture influencing factors related to the video stream include at least one of the following: the resolution of a terminal device used to play the video stream, the state parameters of a network device used to transmit the video stream, and the motion state of a target object in the video stream.

3. The method according to claim 1 or 2, characterized in that: The picture impact factor related to the video stream includes a motion state of a target object in the video stream, and collecting the picture impact factor related to the video stream includes: Obtaining a motion vector according to a first video frame and a second video frame in the video stream; The motion vectors are statistically analyzed, and classification is performed according to the statistical analysis results of the motion vectors to obtain the motion state of the target object in the video stream.

4. The method according to claim 3, characterized in that The statistical analysis result of the motion vector includes at least one of the number, maximum value, mean value, and entropy value of the motion vector, and the classification according to the statistical analysis result of the motion vector includes: When the number of the motion vectors is greater than a first number and less than a second number, and the average of the motion vectors is greater than a first threshold and less than a second threshold, it is determined that the motion state is a normal moving state; When the maximum value of the motion vector is greater than a third threshold, determining that the motion state is a first abnormal movement state; When the mean value of the motion vectors is less than the first threshold, determining that the motion state is a stationary state; When the number of the motion vectors is less than the first number and the average of the motion vectors is greater than the second threshold, determining that the motion state is a second abnormal movement state; When the entropy value of the motion vector is greater than a fourth threshold, the motion state is determined to be a third abnormal movement state.

5. The method according to any one of claims 2 to 4, characterized in that: When the picture impact factor meets the set condition, the video stream is subjected to motion estimation and motion compensation MEMC processing to obtain a processed video stream, including: When the resolution of the terminal device used to play the video stream is greater than the set resolution, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or, When a state parameter of a network device used to transmit the video stream is less than a set value, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or, When the motion state of the target object in the video stream is a normal moving state, motion estimation and motion compensation MEMC processing is performed on the video stream to obtain a processed video stream.

6. The method according to any one of claims 1 to 5, characterized in that: The video stream includes a processed interval, a cache interval and subsequent video frames, the processed interval includes video frames that have been processed by MEMC, the cache interval is used to cache video frames to be processed in the video stream, and the method further includes: When the MEMC state is switched, the video frame cached in the cache interval is used as the starting position, and the frame discarding operation is performed every n frames until the number of dropped frames meets the target value, and the frame discarding operation is stopped.

7. The method according to any one of claims 1 to 6, characterized in that: The video stream to be processed is a conference video stream collected by a conference terminal.

8. A video processing system, characterized in that: The system comprises: A communication module, used for obtaining a video stream to be processed; An analysis and decision module, used for collecting picture impact factors related to the video stream, wherein the picture impact factors include parameters affecting the picture smoothness of the video stream; The motion estimation and motion compensation MEMC module is used to perform MEMC processing on the video stream to obtain a processed video stream when the picture impact factor meets the set conditions.

9. The system according to claim 8, characterized in that The picture influencing factors related to the video stream include at least one of the following: the resolution of a terminal device used to play the video stream, the state parameters of a network device used to transmit the video stream, and the motion state of a target object in the video stream.

10. The system according to claim 8 or 9, characterized in that The picture influencing factors related to the video stream include the motion state of the target object in the video stream, and the analysis and decision module is specifically used for: Obtaining a motion vector according to a first video frame and a second video frame in the video stream; The motion vectors are statistically analyzed, and classification is performed according to the statistical analysis results of the motion vectors to obtain the motion state of the target object in the video stream.

11. The system according to claim 10, characterized in that The statistical analysis result of the motion vector includes at least one of the number, maximum value, mean value, and entropy value of the motion vector, and the analysis and decision module is specifically used for: When the number of the motion vectors is greater than a first number and less than a second number, and the average of the motion vectors is greater than a first threshold and less than a second threshold, it is determined that the motion state is a normal moving state; When the maximum value of the motion vector is greater than a third threshold, determining that the motion state is a first abnormal movement state; When the mean value of the motion vectors is less than the first threshold, determining that the motion state is a stationary state; When the number of the motion vectors is less than the first number and the average of the motion vectors is greater than the second threshold, determining that the motion state is a second abnormal movement state; When the entropy value of the motion vector is greater than a fourth threshold, the motion state is determined to be a third abnormal movement state.

12. The system according to any one of claims 9 to 11, characterized in that The MEMC module is specifically used for: When the resolution of the terminal device used to play the video stream is greater than the set resolution, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or, When a state parameter of a network device used to transmit the video stream is less than a set value, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or, When the motion state of the target object in the video stream is a normal moving state, motion estimation and motion compensation MEMC processing is performed on the video stream to obtain a processed video stream.

13. The system according to any one of claims 8 to 12, characterized in that: The video stream includes a processed interval, a cache interval and subsequent video frames, the processed interval includes video frames that have been processed by MEMC, the cache interval is used to cache video frames to be processed in the video stream, and the system further includes: The state switching processing module is used to, when the MEMC state is switched, take the video frame cached in the cache interval as the starting position, perform frame discarding operation every n frames until the number of dropped frames meets the target value, and stop the frame discarding operation.

14. The system according to any one of claims 8 to 13, characterized in that The video stream to be processed is a conference video stream collected by a conference terminal.

15. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster executes the method as described in any one of claims 1 to 7.

16. A terminal device, characterized in that: The terminal device comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the terminal device executes the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 7.

18. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 7.

Images