Video processing method and related equipment

Abstract

The invention provides a video processing method, which comprises the following steps: acquiring a to-be-processed video stream, acquiring a picture influence factor related to the video stream, performing motion estimation and motion compensation (MEMC) processing on the video stream when the picture influence factor satisfies a set condition, and obtaining a processed video stream. In this way, MEMC self-adaptive adjustment based on the picture influence factor can be realized, and the situation that the video picture is stuck or damaged after the MEMC is started can be effectively solved. Moreover, according to the method, manual operation of the user is not needed, for example, the user does not need to manually switch MEMC parameters through a remote controller, the two hands of the user are liberated, the user does not need to concern on or off of the MEMC, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, a video processing system, a terminal device, a computing device cluster, a computer-readable storage medium, and a computer program product. Background Art

[0002] In order to provide users with a better visual experience, motion estimation and motion compensation (MEMC) came into being. MEMC predicts the motion trajectory of an object and compensates between two frames, compensating low-frame-rate videos to high-frame-rate videos, making the images smoother to watch.

[0003] Currently, many video application scenarios try to use MEMC to improve the smoothness of video images. For example, TV video playback scenarios or resident conference scenarios support turning on the MEMC function to compensate for the video images and improve the visual experience of viewers or participants. However, not all video streams are suitable for turning on the MEMC function.

[0004] The industry has proposed a solution to dynamically adjust MEMC. Taking the video playback scene of a TV as an example, three motion compensation control modes are set inside the main processor of the TV, such as sports mode, game mode, and film mode. The main processor is electrically connected to the MEMC processing module, and the MEMC processing module includes MEMC parameters corresponding to the above three modes. Depending on the content of the video stream played by the TV, the user can transmit a remote control command of the corresponding mode through the remote control. The main processor of the TV receives the remote control command and instructs the MEMC processing module to apply the MEMC parameters corresponding to the mode, so that the picture quality effect (such as smoothness) in different application scenarios can reach the best state.

[0005] However, the above method relies on a pre-set motion compensation control mode and requires the user to manually switch MEMC parameters, which makes it difficult to achieve adaptive adjustment, difficult to meet business needs, and reduces user experience. Summary of the invention

[0006] The present application provides a video processing method, which implements MEMC adaptive adjustment based on the screen influencing factor, and can effectively solve the problem of video freezing or damage after turning on MEMC. In addition, the method does not require manual operation by the user, and the user does not need to care about turning MEMC on or off, which improves the user experience. The present application also provides a video processing system, a computing device cluster, a terminal device, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0007] In the first aspect, the present application provides a video processing method. The method can be performed by a video processing system. The video processing system can be a software system, which can be deployed in a computing device cluster, such as a terminal device, or distributed in a terminal device and a server. For ease of description, the present application is mainly illustrated by the deployment of a software system in a terminal device. The terminal device executes the program code of the software system to execute the video processing method of the present application. In some possible implementations, the video processing system can also be a hardware system, and the hardware system can be a computing device cluster with a video processing function, such as a terminal device with a video processing function, or a terminal device and a server that collaborates to complete video processing. Among them, the terminal device can be an electronic device with a MEMC function, including but not limited to a computer, a television (such as a smart screen TV), a large conference screen, and a smart phone. The MEMC function refers to a terminal device having a MEMC module, which can interpolate low frame rate video into high frame rate video, and then transmit it to a display for display.

[0008] In specific implementation, the video processing system can obtain the video stream to be processed, and then collect the picture impact factor related to the video stream. The picture impact factor includes parameters that affect the picture smoothness of the video stream. When the picture impact factor meets the set conditions, the video processing system can perform motion estimation and motion compensation MEMC processing on the video stream to obtain the processed video stream.

[0009] The method adaptively adjusts MEMC through the collected picture impact factors, which can effectively solve the problem of video freezing or damage after turning on MEMC. In addition, the method does not require manual operation by the user, such as manually switching MEMC parameters through a remote control, freeing the user's hands, and the user does not need to care about turning MEMC on or off, thereby improving the user experience.

[0010] In some possible implementations, the picture influencing factors related to the video stream include at least one of the following: the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream. In this way, an adaptive adjustment method based on the network environment and the screen resolution of the terminal device can be implemented, effectively avoiding freezes and image quality damage caused by the network environment, and avoiding the insignificant effect of turning on MEMC due to a small screen, but instead introducing additional delays.

[0011] In some possible implementations, the picture influencing factors related to the video stream include the motion state of the target object in the video stream. Accordingly, the video processing system can obtain a motion vector based on a first video frame and a second video frame in the video stream, for example, adjacent frames in the video stream, and then the video processing system can perform statistical analysis on the motion vector, perform classification based on the statistical analysis results of the motion vector, and obtain the motion state of the target object in the video stream.

[0012] In this way, an adaptive adjustment method based on motion status can be implemented. By analyzing the motion status of the target object in the video stream, it is decided whether to turn on MEMC. According to the above decision, MEMC is turned on in an appropriate time period to ensure the overall picture quality effect (such as smoothness).

[0013] In some possible implementations, the statistical analysis results of the motion vectors include at least one of the number, maximum value, mean value, and entropy value of the motion vectors. Accordingly, when the number of motion vectors is greater than the first number and less than the second number, and the mean value of the motion vectors is greater than the first threshold value and less than the second threshold value, the video processing system can determine that the motion state is a normal moving state; when the maximum value of the motion vectors is greater than the third threshold value, the video processing system can determine that the motion state is a first abnormal moving state; when the mean value of the motion vectors is less than the first threshold value, the video processing system can determine that the motion state is a stationary state; when the number of motion vectors is less than the first number, and the mean value of the motion vectors is greater than the second threshold value, the video processing system can determine that the motion state is a second abnormal moving state; when the entropy value of the motion vectors is greater than the fourth threshold value, the video processing system can determine that the motion state is a third abnormal moving state.

[0014] This method classifies the motion state by combining the number of motion vectors, the mean of motion vectors, and the entropy value of motion vectors, thereby providing assistance in deciding whether to turn on MEMC.

[0015] In some possible implementations, the picture impact factor satisfies the set conditions that the resolution of the terminal device used to play the video stream is greater than the set resolution, the state parameter of the network device used to transmit the video stream is less than the set value, and / or the motion state of the target object in the video stream is a normal moving state.

[0016] In other words, the video processing system can perform MEMC processing on the video stream to obtain a processed video stream when the resolution of the terminal device used to play the video stream is greater than the set resolution; and / or, the video processing system can perform MEMC processing on the video stream to obtain a processed video stream when the state parameter of the network device used to transmit the video stream is less than a set value; and / or, the video processing system can perform MEMC processing on the video stream to obtain a processed video stream when the motion state of the target object in the video stream is a normal moving state.

[0017] In this way, when the resolution of the terminal device is greater than the set resolution, the status parameter of the network device is less than the set value and / or the motion state of the target object in the video stream is in a normal moving state, MEMC can be adaptively turned on to ensure the overall picture quality when the video stream is played.

[0018] In some possible implementations, the video stream includes a processed interval, a cache interval, and subsequent video frames, the processed interval includes video frames that have been processed by MEMC, and the cache interval is used to cache video frames to be processed in the video stream. When the MEMC state is switched, the video processing system can also use the video frames cached in the cache interval as the starting position, perform frame discarding operations every n frames, until the number of dropped frames meets the target value, and stop the frame discarding operation.

[0019] This method achieves smooth switching of MEMC states by dropping frames evenly at intervals, avoids black screen or freeze caused by directly clearing the cache interval (cache queue), and optimizes the video stream when switching MEMC states.

[0020] In some possible implementations, the video stream to be processed is a conference video stream collected by a conference terminal. This method collects screen impact factors related to the conference video stream and adaptively adjusts the MEMC based on the screen impact factors, effectively solving the problem of conference screen freeze and damage after turning on the MEMC, while freeing the user's hands and not having to worry about the MEMC switch.

[0021] In a second aspect, the present application provides a video processing system. The system comprises:

[0022] A communication module, used for obtaining a video stream to be processed;

[0023] An analysis and decision module, used for collecting picture impact factors related to the video stream, wherein the picture impact factors include parameters affecting the picture smoothness of the video stream;

[0024] The motion estimation and motion compensation MEMC module is used to perform MEMC processing on the video stream to obtain a processed video stream when the picture impact factor meets the set conditions.

[0025] In some possible implementations, the picture influencing factors related to the video stream include at least one of the following: the resolution of a terminal device used to play the video stream, the state parameters of a network device used to transmit the video stream, and the motion state of a target object in the video stream.

[0026] In some possible implementations, the picture influencing factor related to the video stream includes a motion state of a target object in the video stream, and the analysis and decision module is specifically configured to:

[0027] Obtaining a motion vector according to a first video frame and a second video frame in the video stream;

[0028] The motion vectors are statistically analyzed, and classification is performed according to the statistical analysis results of the motion vectors to obtain the motion state of the target object in the video stream.

[0029] In some possible implementations, the statistical analysis result of the motion vector includes at least one of the number, maximum value, mean value, and entropy value of the motion vector, and the analysis and decision module is specifically used to:

[0030] When the number of the motion vectors is greater than a first number and less than a second number, and the average of the motion vectors is greater than a first threshold and less than a second threshold, it is determined that the motion state is a normal moving state;

[0031] When the maximum value of the motion vector is greater than a third threshold, determining that the motion state is a first abnormal movement state;

[0032] When the mean value of the motion vectors is less than the first threshold, determining that the motion state is a stationary state;

[0033] When the number of the motion vectors is less than the first number and the average of the motion vectors is greater than the second threshold, determining that the motion state is a second abnormal movement state;

[0034] When the entropy value of the motion vector is greater than a fourth threshold, the motion state is determined to be a third abnormal movement state.

[0035] In some possible implementations, the MEMC module is specifically used for:

[0036] When the resolution of the terminal device used to play the video stream is greater than the set resolution, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or,

[0037] When a state parameter of a network device used to transmit the video stream is less than a set value, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or,

[0038] When the motion state of the target object in the video stream is a normal moving state, motion estimation and motion compensation MEMC processing is performed on the video stream to obtain a processed video stream.

[0039] In some possible implementations, the video stream includes a processed interval, a cache interval, and subsequent video frames, the processed interval includes video frames that have been MEMC processed, the cache interval is used to cache video frames to be processed in the video stream, and the system further includes:

[0040] The state switching processing module is used to, when the MEMC state is switched, take the video frame cached in the cache interval as the starting position, perform frame discarding operation every n frames until the number of dropped frames meets the target value, and stop the frame discarding operation.

[0041] In some possible implementations, the video stream to be processed is a conference video stream collected by a conference terminal.

[0042] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster performs the video processing method described in the first aspect or any implementation of the first aspect.

[0043] In a fourth aspect, the present application provides a terminal device. The terminal device includes a processor and a memory, wherein the memory stores computer-readable instructions; the processor is used to execute the computer-readable instructions so that the terminal device performs the video processing method as described in the first aspect or any implementation of the first aspect.

[0044] In a fifth aspect, the present application provides a computer-readable storage medium having instructions stored therein, the instructions instructing a terminal device or a computing device cluster to execute the video processing method described in the first aspect or any implementation of the first aspect.

[0045] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a terminal device or a computing device cluster, enables the terminal device or the computing device cluster to execute the video processing method described in the first aspect or any implementation of the first aspect.

[0046] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0048] Figure 1 A schematic diagram of the structure of a video processing system provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of video stream optimization during MEMC state switching provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of a video processing method provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a scenario for adaptively adjusting the application of MEMC in a resident conference based on resolution and network environment provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a scenario for adaptively adjusting the application of MEMC in a resident conference based on motion state provided in an embodiment of the present application;

[0053] Figure 6 A flowchart of a method for processing video during state switching provided by an embodiment of the present application;

[0054] Figure 7 A flowchart of the application of a video processing method provided in an embodiment of the present application in an on-site conference;

[0055] Figure 8 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0058] First, some technical terms involved in the embodiments of the present application are introduced.

[0059] Video refers to dynamic images that are captured, recorded, processed, stored, sent, and reproduced in the form of electrical signals. Video streaming refers to videos that support streaming data transmission. Video streams can be processed as a stable, continuous stream through the network, so terminal devices can play videos without waiting for all video data to be transmitted. Video streams can be pre-generated, for example, the video streams of TV programs can usually be pre-recorded. Video streams can also be collected in real time. For example, in a video conferencing scenario, the conference terminal can collect images in real time through the camera to generate a video stream.

[0060] A video stream usually includes a certain number of images, which are also called frames or video frames. The picture quality (such as picture smoothness) when a video stream is played is usually related to the frame rate of the video stream. Among them, the frame rate is also called the picture update rate. The refresh rate refers to the number of static pictures played per second, so it is also called the number of frames transmitted per second (Frames PerSecond, FPS). Typical frame rates range from 6 or 8 frames per second in the early days to 120 frames per second today. In order to achieve basic visual persistence effects, the frame rate usually needs to reach 10FPS. For example, a common frame rate for movies is 24FPS. Generally speaking, the higher the frame rate, the smoother the picture when the video stream is played.

[0061] Terminal devices that use liquid crystal displays, such as LCD TVs and conference terminals, usually use electric current to stimulate liquid crystal molecules to generate points, lines, and surfaces that form a picture with the back light tube. Liquid crystal molecules only work when they receive signals, and once the picture changes too quickly, it is difficult for the liquid crystal molecules to respond in time. At this time, the terminal device still displays the picture of the previous second, which is intuitively manifested as a blurred picture. In order to improve the above-mentioned picture blur (for example, motion blur), the internal circuit processing technology can be improved to increase the number of video frames displayed per unit time. Correspondingly, the video stream can also perform video processing such as video interpolation to adapt to the terminal device to improve the smoothness of the picture.

[0062] Video interpolation refers to inserting new frames between adjacent frames in a video stream to increase the frame rate of the video stream. Video interpolation includes a variety of implementation methods, such as inserting black frames, inserting duplicate frames, and inserting computational frames. Motion estimation and motion compensation (MEMC) is an example of interpolation computational frames. MEMC automatically generates a new frame between the two original frames by intelligently analyzing the two adjacent frames. This not only makes the picture smoother, but also effectively reduces the phenomenon of image tailing.

[0063] However, not all video streams are suitable for enabling MEMC. MEMC may make calculation errors or fail when facing complex images. For example, MEMC can estimate the motion trajectory of a small number of moving objects, but when more moving objects suddenly appear in the picture, the effect of MEMC will be greatly reduced.

[0064] To this end, most manufacturers provide a switch option for the MEMC function, allowing users to freely choose according to the actual viewing screen. Some manufacturers even provide different gears or modes of MEMC, allowing users to choose the gear or mode that suits the video stream being played. For example, TV manufacturers set a variety of motion compensation control modes inside the main processor of the TV, including but not limited to sports mode, game mode, and film mode. The MEMC processing module electrically connected to the main processor includes MEMC parameters corresponding to the above modes. Depending on the content of the video stream played by the TV, the user can use the remote control to transmit a remote control command of the corresponding mode. The main processor of the TV receives the remote control command and instructs the MEMC processing module to apply the MEMC parameters corresponding to the mode, so that the picture quality effect (such as smoothness) in different application scenarios can reach the best state.

[0065] The above method relies on a pre-set motion compensation control mode and requires the user to manually switch MEMC parameters, which makes it difficult to achieve adaptive adjustment, difficult to meet business needs, and reduces user experience.

[0066] In view of this, the present application provides a video processing method. The method can be performed by a video processing system. The video processing system can be a software system, which can be deployed in a computing device cluster, such as a terminal device, or distributedly deployed in a terminal device and a server. For ease of description, the present application is mainly illustrated by the deployment of a software system in a terminal device. The terminal device executes the program code of the software system to execute the video processing method of the present application. In some possible implementations, the video processing system can also be a hardware system, and the hardware system can be a computing device cluster with a video processing function, such as a terminal device with a video processing function, or a terminal device and a server that collaborates to complete video processing. Among them, the terminal device can be an electronic device with a MEMC function, including but not limited to a computer, a television (such as a smart screen TV), a large conference screen, and a smart phone. The MEMC function refers to a terminal device having a MEMC module, which can interpolate a low frame rate video into a high frame rate video, and then transmit it to a display for display.

[0067] In specific implementation, the video processing system obtains the video stream to be processed and collects the picture impact factor related to the video stream, which includes parameters that affect the picture smoothness of the video stream. When the picture impact factor meets the set conditions, motion estimation and motion compensation MEMC processing is performed on the above video stream to obtain the processed video stream.

[0068] In this way, MEMC adaptive adjustment based on the picture influencing factor can be achieved, which can effectively solve the problem of video freezing or damage after turning on MEMC. In addition, this method does not require manual operation by the user, such as manually switching MEMC parameters through a remote control, freeing the user's hands, and the user does not need to care about turning MEMC on or off, thus improving the user experience.

[0069] The video processing method of the present application can be applied to a variety of video processing scenarios, including but not limited to TV video playback, video live broadcast or video conferencing scenarios. Especially in the scenario of resident conference (users build a complete conference system by themselves and can enjoy conference services for life), by collecting the resolution of conference terminals (such as special conference terminals such as large conference screens, or general conference terminals such as computers and mobile phones that support video conferencing), conference network conditions (such as the status parameters of network devices used to transmit conference video streams), the motion status of target objects in conference video streams and other screen influencing factors, and automatically starting MEMC processing of the video stream when the screen influencing factors meet the conditions, thereby solving the problems of screen freezes and image quality damage caused by poor network status, fast movement of participants in the scene or frequent small-area movements in the scene.

[0070] In order to make the technical solution of the present application clearer and easier to understand, the architecture of the video processing system of the present application is introduced below using an immersive conference scenario.

[0071] See also Figure 1 The structure diagram of a video processing system is shown in FIG. 1 . The video processing system can be deployed in a conference terminal, for example, in Figure 1 The conference terminal A shown in FIG. Figure 1 In the conference terminal B shown. Conference terminal A and conference terminal B can be dedicated conference terminals, such as a large conference screen, or general conference terminals, such as a computer or a mobile phone. When conference terminal A and conference terminal B participate in the same conference, conference terminal B can execute the program code of the video processing system to process the conference video stream sent by conference terminal A. It should be noted that conference terminal A can also execute the program code of the video processing system to process the conference video stream sent by conference terminal B. This embodiment takes the processing of the conference video stream sent by conference terminal A by conference terminal B as an example.

[0072] Specifically, the conference terminal B is deployed with a video processing system 100, which includes a communication module ( Figure 1(not shown), analysis and decision module 102 and MEMC module 104. Among them, the communication module is used to obtain the video stream to be processed, and the video stream to be processed can be a conference video stream, such as a conference video stream generated by real-time acquisition of conference terminal A. The analysis and decision module 102 is used to collect the picture impact factor related to the video stream, and the picture impact factor includes parameters that affect the picture smoothness of the video stream. Among them, the picture impact factor related to the video stream includes at least one of the following: the resolution of the terminal device (for example, conference terminal B) used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream. In some examples, the analysis and decision module 102 may include one or more of the resolution and network status analysis and decision module 102A and the motion state analysis and decision module 102B. The MEMC module 104 is used to perform MEMC processing on the video stream when the picture impact factor meets the set conditions to obtain the processed video stream. In this way, it is possible to adaptively adjust the MEMC using the resolution or network status, or adaptively adjust the MEMC using the motion state of the object in the scene. The video stream processed by the MEMC module 104 can be transmitted to the display for display. Compared with directly displaying the original video stream, displaying the processed video stream can achieve better fluency and provide better visual effects.

[0073] It should be noted that whether the picture impact factor meets the set condition can be determined by the analysis and decision module 102, and a corresponding MEMC decision is generated. The MEMC decision is used to indicate whether to perform MEMC processing (or whether to turn on MEMC).

[0074] Furthermore, MEMC state switching can be generated during the adaptive adjustment of MEMC. For example, as the meeting progresses, the motion state of the participants or other objects (such as clocks) appearing in the meeting screen may change, and accordingly, the MEMC state may also change. The following is explained with examples. In this example, MEMC can be turned on when the participants (for example, participants in the conference video stream collected by conference terminal A) are walking normally; MEMC can be turned off when the participants continue to sit in a fixed position and are in a stationary state. When a MEMC state switch occurs, for example, when the MEMC state switches from an on state to an off state, a momentary black screen or freeze may occur. The present application also supports processing of video streams to solve the black screen or freeze problems caused by MEMC state switching.

[0075] In a specific implementation, the video processing system 100 may further include a state switching processing module 106. After the MEMC is turned on, the video stream may include a processed interval, a cache interval, and subsequent video frames. Figure 2As shown, the processed interval is also called the inserted frame display interval, and the processed interval includes video frames that have been MEMC processed. The cache interval may include video frames to be MEMC processed. The state switching processing module 106 is used to take the video frame cached in the cache interval as the starting position when the MEMC state is switched, and perform a frame discarding operation every n frames until the number of lost frames meets the target value, and stop the frame discarding operation. For example, the state switching processing module 106 can count the length of the cache interval, starting from the first frame of the cache queue, and discard one frame every N frames until the number of lost frames reaches the length of the cache interval, and stop dropping frames. In this way, a smooth transition of the MEMC state can be completed to avoid a momentary black screen or freeze, which affects the visual effect.

[0076] based on Figure 1 The video processing system 100 shown in the figure, the present application also provides a video processing method. The video processing method of the present application is introduced below in conjunction with an embodiment.

[0077] See also Figure 3 A flow chart of a video processing method is shown, the method comprising:

[0078] S302: The video processing system 100 obtains a video stream to be processed.

[0079] The video stream to be processed may be a video stream for which a decision is to be made whether to perform MEMC processing to improve the degree of picture flow. The video stream to be processed may be a video stream recorded in real time, or a pre-generated video stream. For example, when the video processing system 100 is a video processing system deployed in a terminal device such as a television, the video stream to be processed may be a video stream pushed by the server. Depending on the signal source, the video stream pushed by the server may be a pre-generated video stream (such as a video stream of a pre-produced TV series or movie), or a real-time recorded video stream (such as a live stream collected during a live TV broadcast). For another example, when the video processing system 100 is a video processing system deployed in a conference terminal, the video stream to be processed may be a video stream recorded in real time by the conference terminals of other participants.

[0080] S304: The video processing system 100 collects picture impact factors related to the video stream. When the picture impact factors meet the set conditions, S306 is executed.

[0081] The picture impact factor includes parameters that affect the picture smoothness of the video stream. In some possible implementations, the picture impact factor includes the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream (also referred to as the network state) and the motion state of the target object in the video stream.

[0082] Among them, when the resolution of the terminal device is small (the screen is small), the effect of MEMC is usually not very significant, but it will introduce additional delays (such as the delay of MEMC processing), which will affect the smoothness of the picture. If the network environment is poor and the network state is not good, turning on MEMC can cause the jamming to be more obvious. In addition, if the image quality of the video stream is compressed due to poor network status, it will also affect the smoothness of the picture. Even if MEMC is turned on, the improvement of the smoothness of the picture is relatively limited. In addition, the motion state of the target object in the video stream is different, and the degree of improvement of the visual effect of turning on MEMC can also be different. For example, in some motion states, the visual effect (degree of picture flow) of turning on MEMC is relatively limited, and in other states, the visual effect (picture smoothness) of turning on MEMC is more significantly improved. Therefore, the video processing system 100 can collect at least one of the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream to assist in deciding whether to turn on MEMC.

[0083] In specific implementation, the video processing system 100 can determine whether the picture impact factor meets the set conditions, thereby determining whether to turn on MEMC. When the picture impact factor meets the set conditions, MEMC can be turned on to execute S306, and then the processed video stream is transmitted to the display for display to improve the visual effect. When the picture impact factor does not meet the set conditions, MEMC processing is not performed, and the video processing system 100 can transmit the original video stream to the display for display.

[0084] The setting conditions may be set according to business requirements. In some examples, the setting conditions may include at least one of the following:

[0085] The resolution of the terminal device used to play the video stream is greater than the set resolution;

[0086] The status parameter of the network device used to transmit the video stream is less than the set value;

[0087] The motion scene of the target object in the video stream is a normal moving state.

[0088] The normal moving state means that the object is in a moving state and the moving speed is within a set range. In specific implementation, the video processing system 100 can determine whether the object is in a normal moving state based on the motion vector. In some examples, the number and mean of the motion vectors are within a certain range, for example, the number of motion vectors is greater than the first number and less than the second number, and the mean of the motion vectors is greater than the first threshold and less than the second threshold, then it can be determined that the object is in a normal moving state.

[0089] S306 , the video processing system 100 processes the video stream MEMC to obtain a processed video stream.

[0090] Specifically, the video processing system 100 can perform MEMC processing on the video stream frame by frame to obtain a processed video stream. During the MEMC processing, the video stream can be divided into a processed interval, a cache interval, and subsequent video frames. The processed interval includes video frames that have been MEMC processed, and the cache interval is used to cache video frames to be processed in the video stream. The video frames in the processed interval can be transmitted to the display for display in a time sequence. Accordingly, the video processing system 100 can obtain video frames from the cache interval for MEMC processing, and then add the video frames that have been MEMC processed to the processed interval. In addition, the video processing system 100 can obtain video frames from subsequent video frames and add them to the cache interval to continue MEMC processing.

[0091] Based on the above description, it can be seen that the video processing method provided by the embodiment of the present application can realize MEMC adaptive adjustment based on the screen influencing factor, which can effectively solve the problem of video freezing or damage after turning on MEMC. In addition, the method does not require manual operation by the user, for example, the user does not need to manually switch MEMC parameters through the remote control, freeing the user's hands, and the user does not need to care about turning MEMC on or off, thereby improving the user experience.

[0092] The video processing method of the embodiment of the present application is described in detail below using a conference scenario.

[0093] Taking into account the requirements of process and real-time performance, conference scenes usually cannot frequently experience freezes and high delays. To this end, the video processing system 100 can adaptively adjust MEMC based on resolution and network environment.

[0094] Among them, the processing logic of the video processing system 100 for adaptively adjusting MEMC based on resolution and network environment can be: obtaining the resolution (denoted as curScreenResolution) of the current conference terminal (such as conference terminal B); if the resolution of the current conference terminal is lower than the minimum resolution required by MEMC (denoted as minScreenResolution), since turning on MEMC has limited improvement in visual smoothness and will introduce additional delays, in other words, the disadvantages of turning on MEMC at this time outweigh the benefits. Therefore, if the resolution is lower than the above minimum resolution (set resolution), the video processing system 100 can turn off MEMC.

[0095] Furthermore, the video processing system 100 can also obtain the state parameters of the network device used to transmit the video stream, including but not limited to the current network packet loss rate curPacketLossRate, latency curLatency and jitter curDithering, wherein each state parameter is set with a corresponding threshold value, which is usually a set value, for example, it can be set according to the bandwidth (for example, the bandwidth allocated to the meeting in the meeting scenario). In some examples, the thresholds of the packet loss rate packetLossRateThreshold, latencyThreshold, and jitter ditheringThreshold corresponding to the 2M bandwidth meeting are 20%, 150ms, and 50ms respectively. The video processing system 100 can determine whether the packet loss rate curPacketLossRate, latency curLatency and jitter curDithering of the current network are less than the corresponding threshold value (set value). If not, it means that the network environment does not meet the set conditions. If MEMC is turned on, the jamming will be more obvious. Moreover, if the network environment is not good and the image quality of the video stream is compressed, it will be difficult to improve the picture smoothness even if MEMC is turned on. The video processing system 100 can decide to turn off MEMC, that is, when the network environment does not meet the set conditions, MEMC processing will not be performed on the video stream.

[0096] Figure 4 A schematic diagram of a scenario for adaptively adjusting the application of MEMC in a resident conference based on resolution and network environment is shown, which specifically includes the following steps:

[0097] S402, conference terminal B obtains the resolution curScreenResolution of the current terminal device. If the resolution meets the set condition, S404 is executed; if the resolution does not meet the set condition, S408 is executed to output the original conference video stream.

[0098] Among them, the resolution meeting the set condition may be that the resolution is greater than the set resolution (for example, the set minimum resolution). In this case, conference terminal B can continue to analyze whether other screen influencing factors meet the set conditions, and then determine whether to turn on MEMC. The resolution does not meet the set condition may be that the resolution is less than the set resolution (such as the above-mentioned minimum resolution). If MEMC is turned on, the improvement of the screen flow is not significant, but it will increase the delay. Therefore, S408 is executed to turn off MEMC and output the original conference video stream.

[0099] S404: Conference terminal B obtains the status parameters of the current network device. If the status parameters of the current network device meet the set conditions, S406 is executed to output the processed high frame rate conference video stream. If the status parameters of the current network device do not meet the set conditions, S408 is executed to output the original conference video stream.

[0100] The state parameters characterizing the network state include at least one of the packet loss rate curPacketLossRate, the delay curLatency and the jitter curDithering. The state parameters of the current network device satisfying the set conditions may be that the packet loss rate is less than the corresponding threshold, the delay is less than the corresponding threshold, and / or the jitter is less than the corresponding threshold. Among them, the threshold corresponding to the packet loss rate, the threshold corresponding to the delay, and the threshold corresponding to the jitter can be determined according to the bandwidth (curBandwidth) of the current conference. curBandwidth corresponds to a set of thresholds (P0, L0, D0) of the state parameters characterizing the network state. If the packet loss rate curPacketLossRate is less than P0, the delay curLatency is less than L0, and the jitter curDithering is less than D0, it means that the state parameters of the current network device meet the set conditions, and the conference terminal B can execute S406 to turn on MEMC, thereby outputting a high frame rate conference video stream. If the packet loss rate curPacketLossRate is greater than P0, the delay curLatency is greater than L0, or the jitter curDithering is greater than D0, it means that the status parameters of the current network device do not meet the set conditions and the network environment is poor. Conference terminal B can execute S408 to close MEMC and output the original conference video stream.

[0101] S406: Conference terminal B turns on MEMC.

[0102] S408: Conference terminal B closes MEMC.

[0103] Both the high frame rate conference video stream with MEMC output turned on and the original conference video stream with MEMC output turned off can be transmitted to the display for display.

[0104] The method can, based on the parameters of the resident conference equipment (such as the conference terminal) (including but not limited to the screen resolution) and the conference network parameters (including packet loss rate, delay, and jitter) acquired in real time, promptly turn on MEMC in scenarios where MEMC can effectively improve the visual effect, and promptly turn off MEMC in scenarios where the effect is not obvious and delays and freezes are likely to occur, thereby realizing adaptive adjustment of MEMC based on resolution and network status.

[0105] Taking into account that some scenes in the conference are not suitable for turning on MEMC or the visual effect of turning on MEMC is limited, it is also possible to analyze the conference scenes in real time and make timely adjustments to MEMC according to the current scenes.

[0106] In a specific implementation, the video processing system 100 deployed in the conference terminal can obtain a motion vector based on the first video frame and the second video frame in the video stream, for example, adjacent frames in the video stream. The motion vector can effectively represent the direction and magnitude of the motion of the two frames before and after, so further analyzing the motion vector can obtain the motion state. Specifically, the video processing system 100 can perform statistical analysis on the motion vector, classify according to the statistical analysis results of the motion vector, and obtain the motion state of the target object in the video stream.

[0107] Wherein, the statistical analysis result of the motion vector may include at least one of the number, maximum value, mean value, and entropy value of the motion vector. When the number of motion vectors is greater than the first number and less than the second number, the mean value of the motion vector is greater than the first threshold value and less than the second threshold value, the video processing system 100 can determine that the motion state is a normal moving state. When the maximum value of the motion vector is greater than the third threshold value, the video processing system 100 can determine that the motion state is a first abnormal moving state, and the first abnormal moving state can be a fast moving state. When the number of motion vectors is less than the first number, the mean value of the motion vector is greater than the second threshold value, the video processing system 100 can determine that the motion state is a second abnormal moving state, and the second abnormal moving state can be a frequent small-area moving state (such as clock hands moving, participants waving). When the entropy value of the motion vector is greater than the fourth threshold value, the video processing system 100 can determine that the motion state is a third abnormal moving state, and the third abnormal moving state can be an irregular moving state, such as a jumping state. When the mean value of the motion vector is less than the first threshold value, the video processing system 100 can determine that the motion state is a stationary state (for example, participants continue to sit in a fixed position without people walking).

[0108] When estimating motion vectors, the video processing system 100 may use an optical flow method, a full search method, a pyramid search method, a gradient method, or a block matching method. For ease of description, the following example uses the video processing system 100 estimating motion vectors using an optical flow method.

[0109] The working principle of optical flow method is based on the following assumptions:

[0110] (1) The pixel intensity within a scene remains essentially unchanged between adjacent frames.

[0111] (2) Adjacent pixels have similar motion.

[0112] The optical flow estimation process is as follows:

[0113] Select a point P, at time t0, after Δdt, point P will move to P′, and P′ itself and its surroundings have similar brightness values ​​to P. For point P, assuming the coordinates are (x, y), then:

[0114] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0115] Where I(x,y,t) represents the brightness value of point P at time t. After time dt, point P moves a distance of dx and dy. Assuming the movement is very small, it can be expressed using Taylor's formula as:

[0116]

[0117] HOT is a high-order infinitesimal, which can be obtained from the first assumption and Taylor expansion:

[0118]

[0119] Then divide by Δt to get:

[0120] f x u+f y v+f t =0

[0121] in,

[0122]

[0123] The above formula becomes the optical flow equation, where u and v represent the moving speed in two directions, and f x 、f y 、f t Represents the gradient of brightness on three axes. u,v are the calculated optical flow. It is usually difficult to use one equation to solve two unknowns, so the Lucas-Kanade (abbreviated as LK) method can be used to solve this problem:

[0124] Based on the second assumption, all adjacent pixels have the same movement. The LK method uses a 3*3 window, and the 9 pixels in the window satisfy:

[0125] f x u+f y v+f t =0

[0126] Substituting the points into the equation, the problem is transformed into using 9 points to solve two unknowns. Since the equation is an overdetermined equation, the least squares method can be used to solve the optimal solution. The following is the calculated result:

[0127]

[0128] After obtaining the motion vector, the video processing system 100 can count the number of motion vectors motionVectorNum, the maximum value of the motion vector maxMotionVector, and the mean value of the motion vector Or the entropy of the motion vector mv When calculating the mean value, the video processing system 100 may also select the motion vector with the vector length TopK to calculate the mean value. When calculating the entropy value, the video processing system 100 may also select the motion vector with the vector direction TopK to calculate the entropy value. The specific calculation process is as follows:

[0129] maxMotionVector=max(motionVector)

[0130]

[0131] entropy mv =-∑P(motionVector)log P(motionVector)

[0132] Here, k is an integer greater than 1.

[0133] The video processing system 100 can perform classification according to the above statistical analysis results, obtain the motion state of the target object in the video stream, and make a decision on whether to turn on the MEMC according to the motion state.

[0134] When the number and mean of motion vectors are within the set range, as shown below:

[0135] motionVectorNum∈(num1,num2)

[0136]

[0137] Wherein, num1 represents the first number, num2 represents the second number, threMV1 and threMV2 represent the first threshold and the second threshold, respectively. Based on this, the video processing system 100 can determine that the participants are walking normally, indicating that the motion state of the target object is a normal moving state, and accordingly, the video processing system 100 can turn on MEMC.

[0138] When the maximum value of the motion vector is greater than the third threshold, as shown below:

[0139] maxMotionVector>threMV3

[0140] Wherein, threMV3 represents a third threshold value. The video processing system 100 may determine that the participant moves quickly, indicating that the motion state of the target object is a first abnormal motion state, and accordingly, the video processing system 100 may turn off the MEMC.

[0141] When the mean value of the motion vector is (continuously) less than the first threshold, as shown below:

[0142]

[0143] It means that the participants continue to sit in fixed positions without moving around. The video processing system 100 can determine that the motion state of the moving object is static. At this time, the MEMC gain is limited, and additional delay is introduced. The video processing system 100 can turn off MEMC.

[0144] When the number of motion vectors is less than the first number, the mean of the motion vectors is greater than the second threshold, as shown below:

[0145] motionVectorNum <num1

[0146]

[0147] It indicates that there are frequent small-area movements in the scene (the number of motion vectors is less than a certain threshold and the mean is greater than a certain threshold), such as clock hands, participants waving their hands, etc. The video processing system 100 can determine that the motion state of the target object is the second abnormal movement state, and accordingly, the video processing system 100 can turn off MEMC.

[0148] When the entropy value of the motion vector is (continuously) greater than the fourth threshold, specifically as follows:

[0149] entropy mv >threE

[0150] Among them, entropy mv represents the entropy value of the motion vector, threE represents the fourth threshold, and if the entropy value is continuously greater than the fourth threshold, it indicates that there is irregular motion in the scene, such as jumping. The video processing system 100 can determine that the motion state of the target object is the third abnormal movement state, and accordingly, the video processing system 100 can turn off MEMC.

[0151] Figure 5 A schematic diagram of a scenario for adaptively adjusting the application of MEMC in a resident conference based on motion state is shown, which specifically includes the following steps:

[0152] S502: Conference terminal B determines the motion vector of the target object in the video stream according to adjacent frames in the original conference video stream, and then performs statistical analysis on the motion vector to obtain a statistical analysis result of the motion vector.

[0153] Specifically, conference terminal B (for example, the video processing system 100 deployed in conference terminal B) can estimate the motion vectors of the two frames before and after the original conference video stream by the optical flow method, and count the number of motion vectors motionVectorNum, the maximum value of the motion vector maxMotionVector, and the mean value of the Top K motion vectors And the entropy value of the TopK motion vector mv .

[0154] S504, conference terminal B classifies according to the statistical analysis results of the motion vectors to obtain the motion state of the target object in the video stream. When the motion state meets the set conditions, execute S506; when the motion state does not meet the set conditions, execute S508.

[0155] The motion state satisfies the set condition, which means that the motion state is a normal motion state. The motion state does not satisfy the set condition, which means that the motion state is not a normal motion state, for example, the motion state is a stationary state or an abnormal motion state. The abnormal motion state includes but is not limited to the first abnormal motion state, the second abnormal motion state, and the third abnormal motion state.

[0156] The specific implementation of classifying and determining the motion state according to the statistical analysis results of the motion vectors can be found in the above description of the relevant contents, which will not be repeated here.

[0157] S506: Conference terminal B turns on MEMC.

[0158] S508: Conference terminal B closes MEMC.

[0159] Both the high frame rate conference video stream with MEMC output turned on and the original conference video stream with MEMC output turned off can be transmitted to the display for display.

[0160] This method can use information such as motion status to classify scenes, and adjust the MEMC strategy in time according to the scene category (such as motion status category), effectively reducing the negative impact of MEMC in conference scenes.

[0161] Furthermore, MEMC state switching can also occur during the MEMC processing, for example, switching from the MEMC on state to the MEMC off state. Directly clearing the cache queue (the queue formed by the frames in the cache interval) when switching the MEMC state may cause the conference screen to go black or freeze. For this reason, the present application proposes a method of dropping frames at intervals evenly to achieve smooth switching of MEMC states and avoid black screens.

[0162] In a specific implementation, when the state switching processing module 106 of the video processing system 100 detects that the MEMC state flag is changed to off, the cache interval length cacheFrameNum is determined, such as Figure 3As shown, the video processing system 100 can perform frame discarding operations every several frames (for example, 1 frame) starting from the first frame in the cache interval until the number of dropped frames dropFrameNum is equal to the cache interval length cacheFrameNum, that is, the smooth switching of the MEMC state is completed. This switching method effectively solves the black screen or freeze caused by the MEMC state switching.

[0163] Figure 6 A flow chart of a video processing method during state switching is shown, which specifically includes the following steps:

[0164] S602, conference terminal B checks the MEMC state. When the MEMC state is switched, execute S604, and when the MEMC state is not switched, execute S606.

[0165] Among them, the switching of the MEMC state includes switching the MEMC on state to the MEMC off state. In specific implementation, the conference terminal B is deployed with a video processing system 100, and the MEMC module 104 in the video processing system 100 can maintain the MEMC state flag. When the MEMC state changes, the MEMC module 104 can modify the state flag, and accordingly, the state switching processing module 106 can read the state flag to obtain the MEMC state. When the result of the state switching processing module 106 reading the state flag this time is closed, and the result of the last reading of the state flag is open, it means that the MEMC state has switched.

[0166] S604: Conference terminal B performs frame dropping processing on the conference video stream at intervals and uniformly.

[0167] Among them, conference terminal B pre-sets the frame interval for uniform frame loss, which is recorded as intervalFrameNum. In other words, when the interval uniform frame loss process is triggered, one frame can be dropped every intervalFrameNum frames. When conference terminal B detects that the MEMC state has switched from on to off, it can count the current cache interval length cacheFrameNum, and start from the first frame in the cache interval, and drop frames evenly every interval of intervalFrameNum frames. Every time a frame is lost, the frame loss number is dropFrameNum+1, until dropFrameNum=cacheFrameNum stops dropping frames. In this way, a smooth transition of the MEMC state is completed.

[0168] S606: Conference terminal B continues to perform MEMC processing on the conference video stream.

[0169] It should be noted that both the conference video stream output after the interval uniform frame loss processing and the conference video stream output after the MEMC processing can be transmitted to the display for display.

[0170] In order to address the problem of black screen or freeze every time the MEMC state is switched, which seriously affects the visual experience of the participants, this embodiment realizes a smooth transition of the MEMC state, which on the one hand improves the fluency of the conference screen, and on the other hand can also improve the usability of the MEMC adaptive adjustment method.

[0171] Based on the above embodiments, the present application also provides a video processing method. Figure 7 , the method comprises the following steps:

[0172] S702: The video processing system 100 receives a conference video stream.

[0173] S704: The video processing system 100 checks whether the MEMC state is switched. If yes, S706 is executed; if no, S708 is executed.

[0174] The video processing system 100 may read the status flag to determine whether the MEMC status is switched, or the video processing system 100 may detect a status change event to determine whether the MEMC status is switched.

[0175] S706: The video processing system 100 performs interval uniform frame loss processing on the conference video stream.

[0176] The specific implementation of the video processing system 100 performing the interval uniform frame loss processing on the conference video stream can refer to the above description of the relevant content. After the video processing system 100 performs the interval uniform frame loss processing on the conference video stream, such as the high frame rate conference video stream, it continues to execute S708 to determine whether to perform MEMC processing on the subsequent video frames.

[0177] S708: The video processing system 100 obtains the resolution of the conference terminal used to play the video stream.

[0178] S710: The video processing system 100 determines whether the resolution meets the set condition. If the resolution meets the set condition, the video processing system executes S712; if the resolution does not meet the set condition, the video processing system 100 executes S722.

[0179] S712: The video processing system 100 obtains status parameters of a network device used to transmit a conference video stream.

[0180] S714: The video processing system 100 obtains and determines whether the state parameters of the network device transmitting the conference video stream meet the set conditions. If yes, execute S716; if not, execute S722.

[0181] S716: The video processing system 100 determines the motion state of the target object in the video stream according to adjacent video frames in the conference video stream.

[0182] S718: The video processing system 100 determines whether the motion state meets the set condition. If yes, execute S720; if not, execute S722.

[0183] S720: The video processing system 100 turns on the MEMC.

[0184] In specific implementation, the video processing system 100 turns on MEMC, and after performing MEMC processing on the conference video stream, a high frame rate conference video stream can be output. For example, if the original video conference stream is a 30-frame video conference stream, the video processing system 100 can output a 60-frame video conference stream after turning on MEMC.

[0185] S722: The video processing system 100 turns off the MEMC.

[0186] In specific implementation, the video processing system 100 turns off MEMC and can output the original conference video stream. It should be noted that if the MEMC state is switched from MEMC on to MEMC off, S724 can be executed, and when the state switch is detected, the original conference video stream is continued to be output after the interval uniform frame loss processing is performed. For example, if the original video conference stream is a 30-frame video conference stream, the video processing system 100 can output a 30-frame video conference stream after turning off MEMC.

[0187] S724: The video processing system 100 updates the status flag.

[0188] Whether MEMC is turned on or off, the video processing system 100 can update the status flag to determine whether a MEMC state switch occurs based on the status flag, thereby determining the processing of subsequent video frames. When MEMC is turned on, the video processing system 100 can update the status flag to on, and when MEMC is turned off, the video processing system 100 can update the status flag to off.

[0189] It should also be noted that the above S708 to S718 are a specific implementation of the video processing system 100 obtaining the picture impact factor related to the conference video stream and determining whether the picture impact factor meets the set conditions. In other possible implementation methods of the embodiment of the present application, the video processing system 100 can also obtain the picture impact factor according to other time sequences. For example, the picture impact factor can be obtained in parallel to determine whether each picture impact factor meets the corresponding set conditions, or the video processing system 100 can also obtain other picture impact factors and decide whether to turn on MEMC based on whether other picture impact factors meet the corresponding set conditions.

[0190] The present application also provides a video processing system 100, such as Figure 1 As shown, including:

[0191] Communication Module( Figure 1 ), used to obtain the video stream to be processed.

[0192] An analysis and decision module 102, configured to collect picture impact factors related to the video stream, wherein the picture impact factors include parameters that affect the picture smoothness of the video stream;

[0193] The MEMC module 104 is used to perform MEMC processing on the video stream to obtain a processed video stream when the picture impact factor meets a set condition.

[0194] Exemplarily, the above-mentioned communication module, analysis and decision module 102, and MEMC module 104 may be implemented by hardware, or may be implemented by software.

[0195] When implemented by software, the communication module, the analysis and decision module 102, and the MEMC module 104 can be applications running on a computing device, such as a computing engine. The application can be provided to users in the form of virtualized services. Virtualized services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service can be a service that virtualizes a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) through virtualization technology to provide users with VMs for use on demand. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS for use on demand. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers for use on demand. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable, and its computing performance is no different from that of a traditional physical machine, and it has the characteristics of secure physical isolation. Containers are a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are merely specific examples. In actual applications, the virtualization service may also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0196] When implemented by hardware, the communication module may be a transceiver or a transceiver unit, and the analysis and decision module 102 and the MEMC module 104 may include at least one computing device, such as a server, etc. Alternatively, the analysis and decision module 102 and the MEMC module 104 may also be devices implemented using application-specific integrated circuits (ASICs) or programmable logic devices (PLDs). The PLDs may be complex programmable logical devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof.

[0197] In some possible implementations, the image influencing factors related to the video stream include at least one of the following: the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream. The analysis and decision module 102 may include a resolution and network state analysis and decision module 102A, which is specifically used to collect the resolution of the terminal device used to play the video stream, the state parameters of the network device used to transmit the video stream, and the motion state of the target object in the video stream.

[0198] In some possible implementations, the image influencing factor related to the video stream includes the motion state of the target object in the video stream. The analysis and decision module 102 may include a motion state analysis and decision module 102B. The motion state analysis and decision module 102B is specifically used to: obtain a motion vector according to the first video frame and the second video frame in the video stream; perform statistical analysis on the motion vector, classify according to the statistical analysis result of the motion vector, and obtain the motion state of the target object in the video stream.

[0199] In some possible implementations, the statistical analysis result of the motion vector includes at least one of the number, maximum value, mean value, and entropy value of the motion vector. The analysis and decision module 102 (for example, the motion state analysis and decision module 102B) is specifically used to: when the number of the motion vectors is greater than the first number and less than the second number, and the mean value of the motion vectors is greater than the first threshold value and less than the second threshold value, determine that the motion state is a normal moving state; when the maximum value of the motion vectors is greater than the third threshold value, determine that the motion state is a first abnormal moving state; when the mean value of the motion vectors is less than the first threshold value, determine that the motion state is a stationary state; when the number of the motion vectors is less than the first number, and the mean value of the motion vectors is greater than the second threshold value, determine that the motion state is a second abnormal moving state; when the entropy value of the motion vectors is greater than the fourth threshold value, determine that the motion state is a third abnormal moving state.

[0200] In some possible implementations, the MEMC module 104 is specifically used for:

[0201] When the resolution of the terminal device used to play the video stream is greater than the set resolution, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or,

[0202] When a state parameter of a network device used to transmit the video stream is less than a set value, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or,

[0203] When the motion state of the target object in the video stream is a normal moving state, motion estimation and motion compensation MEMC processing is performed on the video stream to obtain a processed video stream.

[0204] In some possible implementations, the video stream includes a processed interval, a cache interval, and subsequent video frames, the processed interval includes video frames that have been MEMC processed, the cache interval is used to cache video frames to be processed in the video stream, and the system 100 further includes:

[0205] The state switching processing module 106 is used to, when the MEMC state is switched, take the video frame cached in the cache interval as the starting position, perform frame discarding operation every n frames until the number of dropped frames meets the target value, and stop the frame discarding operation.

[0206] Similar to the analysis and decision module 102 and the MEMC module 104 , the state switching processing module 106 may be implemented by hardware or software.

[0207] When implemented by software, the state switching processing module 106 may be an application running on a computing device, such as a computing engine, etc. The application may be provided to users in the form of a virtualized service. The virtualized service may include a VM service, a BMS service, and a container service.

[0208] When implemented by hardware, the state switching processing module 106 may include at least one computing device, such as a server, etc. Alternatively, the state switching processing module 106 may also be a device implemented by ASIC or PLD, etc. The PLD may be implemented by CPLD, FPGA, GAL or any combination thereof.

[0209] In some possible implementations, the video stream to be processed is a conference video stream collected by a conference terminal.

[0210] The present application also provides a terminal device 800. Figure 8 As shown, the terminal device 800 includes: a bus 802, a processor 804, a memory 806, a display 807 and a communication interface 808. The processor 804, the memory 806 and the communication interface 808 communicate with each other through the bus 802. It should be understood that the present application does not limit the number of processors and memories in the terminal device 800.

[0211] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 802 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the terminal device 800 (for example, the memory 806, the processor 804, and the communication interface 808).

[0212] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). In some examples, the terminal device 800 may include a general-purpose CPU, through which the video stream is subjected to MEMC processing. In other examples, the terminal device 800 may include a dedicated MEMC processor (or MEMC chip, MEMC processing module), and the general-purpose CPU and the dedicated MEMC processor collaborate to complete the MEMC processing of the video stream.

[0213] The memory 806 may include a volatile memory, such as a random access memory (RAM). The memory 806 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 806 stores executable program code, and the processor 804 executes the executable program code to implement the aforementioned video processing method. Specifically, the memory 806 stores instructions for the video processing system 100 to execute the video processing method.

[0214] The display 807 is an output device for displaying images and colors. Common displays may be computer or television screens. Depending on the display principle, the display 807 may be of different types. For example, the display 907 may be a liquid crystal display. The display 807 may receive a video stream processed by the processor 804 (such as a general-purpose CPU or a dedicated MEMC processor) and display the image of the processed video stream. Alternatively, the display 807 may also receive the original video stream provided by the processor 804 and display the image of the original video stream.

[0215] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the terminal device 800 and other devices or a communication network.

[0216] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device, for example, a terminal device and at least one server. The server may be a central server, an edge server, or a local server in a local data center.

[0217] like Fig. 9 As shown, the computing device cluster includes Figure 8 The terminal device 800 and at least one server 900 are shown. The server 900 may include a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. It should be understood that the present application does not limit the number of processors and memories in the server 900.

[0218] The memory of the terminal device 800 and the server 900 in the computing device cluster may store the same video processing system 100 for executing instructions of the video processing method.

[0219] In some possible implementations, the terminal device 800 and the server 900 in the computing device cluster can also be used to execute some instructions of the video processing system 100 for executing the video processing method. In other words, the combination of the terminal device 800 and the server 900 can jointly execute the instructions of the video processing system 100 for executing the video processing method.

[0220] It should be noted that the memories in different devices in the computing device cluster may store different instructions for executing partial functions of the video processing system 100 .

[0221] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a terminal device or a computing device cluster to execute the above-mentioned video processing system 100 for executing the video processing method.

[0222] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device cluster or a terminal device or stored in any available medium. When the computer program product is run on a computing device cluster or a terminal device, the computing device cluster or the terminal device executes the above-mentioned video processing method.

[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video processing method, characterized in that: The method comprises: Get the video stream to be processed; Collecting picture impact factors related to the video stream, wherein the picture impact factors include parameters that affect the picture smoothness of the video stream; When the picture impact factor meets the set condition, the video stream is subjected to motion estimation and motion compensation MEMC processing to obtain a processed video stream.

2. The method according to claim 1, characterized in that: The picture influencing factors related to the video stream include at least one of the following: the resolution of a terminal device used to play the video stream, the state parameters of a network device used to transmit the video stream, and the motion state of a target object in the video stream.

3. The method according to claim 1 or 2, characterized in that: The picture impact factor related to the video stream includes a motion state of a target object in the video stream, and collecting the picture impact factor related to the video stream includes: Obtaining a motion vector according to a first video frame and a second video frame in the video stream; The motion vectors are statistically analyzed, and classification is performed according to the statistical analysis results of the motion vectors to obtain the motion state of the target object in the video stream.

4. The method according to claim 3, characterized in that The statistical analysis result of the motion vector includes at least one of the number, maximum value, mean value, and entropy value of the motion vector, and the classification according to the statistical analysis result of the motion vector includes: When the number of the motion vectors is greater than a first number and less than a second number, and the average of the motion vectors is greater than a first threshold and less than a second threshold, it is determined that the motion state is a normal moving state; When the maximum value of the motion vector is greater than a third threshold, determining that the motion state is a first abnormal movement state; When the mean value of the motion vectors is less than the first threshold, determining that the motion state is a stationary state; When the number of the motion vectors is less than the first number and the average of the motion vectors is greater than the second threshold, determining that the motion state is a second abnormal movement state; When the entropy value of the motion vector is greater than a fourth threshold, the motion state is determined to be a third abnormal movement state.

5. The method according to any one of claims 2 to 4, characterized in that: When the picture impact factor meets the set condition, the video stream is subjected to motion estimation and motion compensation MEMC processing to obtain a processed video stream, including: When the resolution of the terminal device used to play the video stream is greater than the set resolution, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or, When a state parameter of a network device used to transmit the video stream is less than a set value, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or, When the motion state of the target object in the video stream is a normal moving state, motion estimation and motion compensation MEMC processing is performed on the video stream to obtain a processed video stream.

6. The method according to any one of claims 1 to 5, characterized in that: The video stream includes a processed interval, a cache interval and subsequent video frames, the processed interval includes video frames that have been processed by MEMC, the cache interval is used to cache video frames to be processed in the video stream, and the method further includes: When the MEMC state is switched, the video frame cached in the cache interval is used as the starting position, and the frame discarding operation is performed every n frames until the number of dropped frames meets the target value, and the frame discarding operation is stopped.

7. The method according to any one of claims 1 to 6, characterized in that: The video stream to be processed is a conference video stream collected by a conference terminal.

8. A video processing system, characterized in that: The system comprises: A communication module, used for obtaining a video stream to be processed; An analysis and decision module, used for collecting picture impact factors related to the video stream, wherein the picture impact factors include parameters affecting the picture smoothness of the video stream; The motion estimation and motion compensation MEMC module is used to perform MEMC processing on the video stream to obtain a processed video stream when the picture impact factor meets the set conditions.

9. The system according to claim 8, characterized in that The picture influencing factors related to the video stream include at least one of the following: the resolution of a terminal device used to play the video stream, the state parameters of a network device used to transmit the video stream, and the motion state of a target object in the video stream.

10. The system according to claim 8 or 9, characterized in that The picture influencing factors related to the video stream include the motion state of the target object in the video stream, and the analysis and decision module is specifically used for: Obtaining a motion vector according to a first video frame and a second video frame in the video stream; The motion vectors are statistically analyzed, and classification is performed according to the statistical analysis results of the motion vectors to obtain the motion state of the target object in the video stream.

11. The system according to claim 10, characterized in that The statistical analysis result of the motion vector includes at least one of the number, maximum value, mean value, and entropy value of the motion vector, and the analysis and decision module is specifically used for: When the number of the motion vectors is greater than a first number and less than a second number, and the average of the motion vectors is greater than a first threshold and less than a second threshold, it is determined that the motion state is a normal moving state; When the maximum value of the motion vector is greater than a third threshold, determining that the motion state is a first abnormal movement state; When the mean value of the motion vectors is less than the first threshold, determining that the motion state is a stationary state; When the number of the motion vectors is less than the first number and the average of the motion vectors is greater than the second threshold, determining that the motion state is a second abnormal movement state; When the entropy value of the motion vector is greater than a fourth threshold, the motion state is determined to be a third abnormal movement state.

12. The system according to any one of claims 9 to 11, characterized in that: The MEMC module is specifically used for: When the resolution of the terminal device used to play the video stream is greater than the set resolution, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or, When a state parameter of a network device used to transmit the video stream is less than a set value, performing motion estimation and motion compensation MEMC processing on the video stream to obtain a processed video stream; and / or, When the motion state of the target object in the video stream is a normal moving state, motion estimation and motion compensation MEMC processing is performed on the video stream to obtain a processed video stream.

13. The system according to any one of claims 8 to 12, characterized in that The video stream includes a processed interval, a cache interval and subsequent video frames, the processed interval includes video frames that have been processed by MEMC, the cache interval is used to cache video frames to be processed in the video stream, and the system further includes: The state switching processing module is used to, when the MEMC state is switched, take the video frame cached in the cache interval as the starting position, perform frame discarding operation every n frames until the number of dropped frames meets the target value, and stop the frame discarding operation.

14. The system according to any one of claims 8 to 13, characterized in that The video stream to be processed is a conference video stream collected by a conference terminal.

15. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster executes the method as described in any one of claims 1 to 7.

16. A terminal device, characterized in that: The terminal device comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the terminal device executes the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 7.

18. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 7.

Images