Video call method and system

Abstract

The invention relates to a video call method and system. The method comprises the following steps: acquiring a video stream of a video call, and extracting all video frames from the video stream; determining a key video frame and a non-key video frame corresponding to the key video frame from all the video frames based on a dynamic vector detection algorithm; carrying out downsampling processing on the non-key video frame; and sending the non-key video frames after downsampling processing and the corresponding key video frames to an intelligent terminal. According to the invention, the problem that resource finiteness of equipment cannot be considered while video call quality is improved in the prior art is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a video calling method and system. Background Art

[0002] With the development of communication technology, video call functions are gradually applied to wearable devices such as smart watches to improve the convenience of communication between people. However, due to the limited hardware resources of wearable devices such as smart watches, huge amounts of traffic are often generated during video calls, which affects the battery life of the device.

[0003] To solve the above problems, the existing technologies usually adopt video compression, expand the battery capacity of the device, or increase the processing power of the relevant integrated chip in the device. However, these methods cannot improve the quality of video calls while taking into account the limited resources of the device. Therefore, it is necessary to provide a video call method and system. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a video call method and system, which improves the problem that the prior art cannot improve the quality of video calls while taking into account the limited resources of the device.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides a video calling method, which is applied to a wearable device, and the method includes: obtaining a video stream of a video call and extracting all video frames from the video stream; based on a dynamic vector detection algorithm, determining key video frames and corresponding non-key video frames from all the video frames; downsampling the non-key video frames; and sending the downsampled non-key video frames and the corresponding key video frames to a smart terminal.

[0006] In one embodiment of the present invention, the method of determining key video frames and corresponding non-key video frames from all video frames based on a dynamic vector detection algorithm includes: preprocessing all video frames; analyzing all preprocessed video frames, and analyzing each video frame as a target video frame: obtaining a first differential image between the target video frame and a first adjacent video frame based on the difference in pixels between the target video frame and the first adjacent video frame; wherein the first video frame is located before the target video frame; obtaining a second differential image between the target video frame and a second adjacent video frame based on the difference in pixels between the target video frame and the second adjacent video frame; wherein the second video frame is located after the target video frame; and determining whether the target video frame is a key video frame or a non-key video frame based on the first differential image and the second differential image.

[0007] In one embodiment of the present invention, the target video frame is determined to be a key video frame or a non-key video frame based on the first differential image and the second differential image: the first differential image and the second differential image are logically ANDed to obtain a difference composite image; the difference composite image is binarized to obtain a binary image; connectivity analysis is performed on the binary image to identify all connected regions in the binary image; based on the areas of all connected regions, the target video frame is determined to be a key video frame or a non-key video frame.

[0008] In one embodiment of the present invention, the determination of whether the target video frame is a key video frame or a non-key video frame based on the areas of all connected regions includes: analyzing each connected region one by one: determining whether the area of ​​the connected region is greater than a preset area threshold: if so, the target video frame is the key video frame of the connected region; otherwise, continuing to analyze the next connected region; after all connected regions are analyzed, if the target video frame is not the key video frame of any connected region, the target video frame is treated as a non-key video frame.

[0009] In one embodiment of the present invention, a video call method is also provided, which is applied to a smart terminal. The method includes: receiving key video frames and corresponding non-key video frames sent by a wearable device; inputting the key video frames and the non-key video frames into a lightweight super-resolution model, and performing super-resolution restoration on the non-key video frames based on the key video frames; arranging the key video frames and the restored non-key video frames in sequence, and presenting them as target videos.

[0010] In one embodiment of the present invention, the key video frame and the non-key video frame are input into a lightweight super-resolution model, and the non-key video frame is super-resolution restored according to the key video frame, including: inputting the key video frame into a first feature extraction module of the lightweight super-resolution model, extracting features from the key frame video, and obtaining a key frame feature map; inputting the non-key video frame into a second feature extraction module of the lightweight super-resolution model, extracting features from the non-key video frame, and obtaining a non-key frame feature map; inputting the key frame feature map and the non-key frame feature map into the lightweight super-resolution model. A feature fusion module of the type is used to fuse the key frame feature map with the non-key frame feature map to obtain a fused feature map; the fused feature map is input into the basic residual module group of the lightweight super-resolution model, feature processing is performed through each basic residual module to obtain a residual feature map, and all the residual feature maps are subjected to multi-level residual mapping to obtain an enhanced feature map; wherein the basic residual module group includes a plurality of cascaded basic residual modules; the enhanced feature map is input into the upsampling module of the lightweight super-resolution model, the enhanced feature map is upsampled, and a non-key video frame after super-resolution restoration is obtained.

[0011] In one embodiment of the present invention, the feature processing is performed through each basic residual module to obtain a residual feature map, including: inputting the fused feature map / residual feature map into the feature extraction unit of the corresponding basic residual module for feature extraction to obtain a depth feature map; inputting the depth feature map and the fused feature map / residual feature map into the residual connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

[0012] In one embodiment of the present invention, the feature processing is performed through each basic residual module to obtain a residual feature map, including: inputting the fused feature map / residual feature map into the first feature extraction unit of the corresponding basic residual module to extract the first feature, and based on the pixel attention mechanism, the extracted first feature is enhanced to obtain an enhanced feature map; inputting the fused feature map / residual feature map into the second feature extraction unit of the corresponding basic residual module to extract the second feature, and processing the second feature on the basis of retaining the features of the fused feature map / residual feature map to obtain a basic feature map; inputting the enhanced feature map and the basic feature map into the splicing unit of the corresponding basic residual module for feature splicing to obtain a comprehensive feature map; inputting the comprehensive feature map and the input fused feature map / residual feature map into the connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

[0013] In one embodiment of the present invention, the feature processing is performed through each basic residual module to obtain a residual feature map, including: inputting the fused feature map / residual feature map into the feature enhancement unit of the corresponding basic residual module, extracting features layer by layer through multiple convolutional layers, and performing residual connection with the input features of the corresponding convolutional layer after each feature extraction to obtain an enhanced feature map; inputting the fused feature map / residual feature map and the enhanced feature map into the connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

[0014] In one embodiment of the present invention, a video call system is also provided, the system comprising a wearable device end and a smart terminal that are communicatively connected to each other: the wearable device end comprises: a video frame extraction module, used to obtain a video stream of a video call and extract all video frames from the video stream; a video frame screening module, used to determine key video frames and corresponding non-key video frames from all video frames based on a dynamic vector detection algorithm; a downsampling module, used to downsample the non-key video frames; a first communication module, used to send the non-key video frames and the corresponding key video frames after downsampling to the smart terminal; the smart terminal comprises: a second communication module, used to receive the key video frames and the corresponding non-key video frames sent by the wearable device end; a super-resolution restoration module, used to input the key video frames and the non-key video frames into a lightweight super-resolution model, and perform super-resolution restoration on the non-key video frames according to the key video frames; a presentation module, used to arrange the key video frames and the restored non-key video frames in sequence and present them as target videos.

[0015] As described above, a video call method and system of the present invention has the following beneficial effects: the wearable device extracts key video frames and non-key video frames from the video stream based on the dynamic vector detection algorithm, and downsamples the non-key video frames. This greatly reduces the amount of data transmission, helps reduce the traffic consumption and power consumption of the wearable device during video calls, and extends its battery life. After the smart terminal receives the non-key video frames and corresponding key video frames after downsampling processing transmitted by the wearable device, it uses the lightweight super-resolution model deployed by itself to perform super-resolution restoration of the non-key video frames based on the key video frames, which can make the restored non-key video frames close to the original resolution, ensuring the real-time and quality of the video call. The present invention makes full use of the strong computing power of the smart terminal, reduces the computing burden of the wearable device, and can not only effectively improve the battery life of the wearable device and reduce its power consumption, but also ensure that the quality of the video call is not affected. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1Shown is a schematic diagram of the overall process of the video calling method provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram showing a flow chart of a video calling method on a wearable device provided by an embodiment of the present invention;

[0018] Figure 3 A schematic diagram showing a flow chart of an ISP chip processing process provided by an embodiment of the present invention;

[0019] Figure 4 Displaying a first differential image and a second differential image provided by an embodiment of the present invention;

[0020] Figure 5 Shown is a schematic diagram of a key frame extraction process provided by an embodiment of the present invention;

[0021] Figure 6 A schematic diagram showing downsampling provided by an embodiment of the present invention is shown;

[0022] Figure 7 A schematic diagram showing a flow chart of a video calling method for a smart terminal provided by an embodiment of the present invention;

[0023] Figure 8 Shown is a schematic diagram of a process of super-resolution restoration provided by an embodiment of the present invention;

[0024] Fig. 9 A model structure diagram showing a basic residual module provided by an embodiment of the present invention;

[0025] Fig.10 A model structure diagram showing a basic residual module based on channel segmentation and attention mechanism provided by an embodiment of the present invention;

[0026] Fig.11 Shown is a model structure diagram of a basic residual module based on channel segmentation provided by an embodiment of the present invention;

[0027] Fig.12 A model structure diagram of a basic residual module based on a shallow residual block provided by an embodiment of the present invention is shown;

[0028] Fig.13 Display as Fig.12 Model structure diagram of the middle and shallow residual blocks;

[0029] Fig.14 A model structure diagram of a basic residual module based on a shallow residual block and channel segmentation provided by an embodiment of the present invention is shown;

[0030] Fig.15 Shown is a comparative schematic diagram of super-resolution restoration provided by an embodiment of the present invention;

[0031] Fig.16 Shown is a structural block diagram of a video call system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0033] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0034] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0035] The inventors found that with the continuous development of communication technology, the use of wearable devices such as smart watches for video calls often generates huge amounts of traffic. In addition, the watch will also have problems such as high performance consumption and reduced battery life during the transmission process. Currently, in order to address the above problems, the industry usually uses video compression technology to compress the video to a certain capacity for transmission, or by expanding the battery capacity, increasing the processing of related integrated chips on wearable devices such as smart watches, etc., to alleviate the above problems to a certain extent.

[0036] In specific implementation, 1) when using the video compression transmission method, the generated video needs to be compressed on the smart watch side, and the video needs to be transmitted to the mobile phone through a local area network / cellular network such as WiFi, and the mobile phone side receives the video transmission content and decompresses and displays it. 2) When expanding the battery capacity, the area occupied by the battery is expanded by compressing the area of ​​other hardware in the smart watch. 3) When increasing the processing power of the integrated chip, it is mainly achieved by replacing the CPU with a more excellent performance and adding a GPU module with strong image rendering capabilities.

[0037] However, these processing methods all have their own disadvantages: 1) For video compression, on the one hand, the compression and decompression methods based on traditional methods cannot restore the original video well, resulting in a decrease in the quality of the video presented by the mobile phone. On the other hand, most of the video compression and decompression algorithms currently used on watches use more general encoding and decoding algorithms, rather than being specially designed for the watch video call scenario. They do not take into account that in most scenarios, the video calls made with watches are made with smart terminal devices with relatively strong performance, such as mobile phones and tablets, and the computing power of these smart terminal devices with relatively strong performance is not fully utilized to reduce the flow and power consumption of the watch. 2) For the method of expanding the battery capacity, on the one hand, it is often necessary to occupy other hardware areas, resulting in the weakening of the watch's capabilities in other aspects. On the other hand, expanding the battery capacity often leads to an increase in the overall weight of the watch, reducing the portability of the watch. 3) For the method of increasing the processing power of the integrated chip, on the one hand, high-performance chips are often more expensive. On the other hand, enhancing the integrated chip on the watch side only reduces the performance consumption during video generation and display, and does not reduce the power consumption during transmission.

[0038] In order to improve the above problems, the present invention provides a video calling method, which is aimed at the phenomenon that the performance of smart terminals such as mobile phones in the current market is basically in excess, but the performance of wearable devices such as smart watches is insufficient. By using the terminal computing power to deploy a lightweight super-resolution network in the smart terminal, the power consumption of the wearable device is optimized, and the overall battery life of the wearable device is greatly improved. In addition, considering that the super-resolution algorithm needs to balance the relationship between image quality and super-resolution network inference speed during image restoration, the lightweight super-resolution algorithm proposed in the present invention ensures the real-time performance of the entire video call process and takes into account the image quality.

[0039] like Figure 1As shown, the present invention makes full use of the computing power of the smart terminal (mainly mobile phone) for video calls with wearable devices, thereby reducing the power consumption and traffic consumption of wearable devices such as watches. Moreover, due to the lightweight network architecture used, the real-time performance during the video call is not affected. The overall process of the present invention is: when making a video call, the video data is collected by the image hardware sensor installed in the wearable device. The collected video data is processed by the ISP (Image Signal Processor) chip in the wearable device to complete the effect processing of the digital image and generate a digital image that is more in line with human vision. These processes mainly include bad pixel correction, denoising, strong light suppression, backlight compensation, color enhancement, lens shading correction, etc. Among them, the MVD (Motion Vector Detection) module in the ISP chip is mainly used to detect dynamic vector information, which can assist us in outputting key video frame information in the video stream. Then, the video stream processed by the ISP chip is downsampled frame by frame except for the key video frame. After processing, the wearable device sends the downsampled low-resolution video to the smart terminal through wireless transmission, which greatly reduces the consumption of traffic and performance during video transmission. By deploying a lightweight super-resolution algorithm on smart terminals such as mobile phones and tablets, after receiving the downsampled video stream, the lightweight super-resolution algorithm is used to restore the resolution of the original video and then display it on the smart terminal, thus realizing the complete video call processing process.

[0040] Specifically, Figure 2 As shown, the video calling method is applied to a wearable device, and the method comprises the following steps:

[0041] S201. Obtain a video stream of a video call, and extract all video frames from the video stream.

[0042] The wearable device responds to the video call request, and after the video call is connected, the image hardware sensor such as the camera installed on the wearable device captures real-time video data and generates a continuous video stream. Since the video stream is composed of a series of continuous video frames, the wearable device extracts all video frames from the video stream at a preset frame rate by extracting them frame by frame to form a video frame sequence. Among them, wearable devices include but are not limited to smart watches, smart glasses, virtual reality headsets and other devices, as long as they can make video calls, and are not limited here.

[0043] S202: Determine key video frames and corresponding non-key video frames from all the video frames based on a dynamic vector detection algorithm.

[0044] The wearable device uses a dynamic vector detection algorithm to analyze each video frame in the video stream frame by frame. For the current frame analyzed, the motion amplitude between the frame and the adjacent frames is analyzed to determine whether the frame is a key video frame. Among them, a key video frame refers to a video frame with a motion amplitude greater than a preset threshold, and each video frame between two key video frames is regarded as a non-key video frame of the previous key video frame.

[0045] Specifically, in one embodiment of the present invention, the step of determining the key video frames and the corresponding non-key video frames from all the video frames based on the dynamic vector detection algorithm includes:

[0046] Preprocessing all the video frames;

[0047] Analyze all the preprocessed video frames, and analyze each video frame as a target video frame:

[0048] Obtaining a first differential image between the target video frame and a first video frame adjacent thereto based on the difference in pixel points between the target video frame and the first video frame adjacent thereto; wherein the first video frame is located before the target video frame;

[0049] Based on the difference of the pixel points between the target video frame and a second video frame adjacent thereto, a second differential image between the two is obtained; wherein the second video frame is located after the target video frame;

[0050] Based on the first differential image and the second differential image, it is determined whether the target video frame is a key video frame or a non-key video frame.

[0051] like Figure 3As shown, after the video frame is converted into a Bayer array, it is input into the ISP chip, wherein the Bayer array refers to a common method used when a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) device is used as a light sensor to collect digital images. The ISP chip in the wearable device performs image signal conversion on all acquired video frames, and pre-processes each video frame through a format processing module to complete the effect processing of the digital image, wherein the ISP chip mainly refers to an image signal processing chip, and a multi-module image processing algorithm is built into the chip for various processing of the image. Further, the pre-processing process includes but is not limited to bad pixel correction, denoising, strong light suppression, backlight compensation, color enhancement, lens shading correction, etc. Through pre-processing, a RAW (Raw Image Data) digital image that is more in line with human vision can be generated. After the pre-processed video stream flows into the MVD module built into the ISP chip, the key video frames in the video stream are extracted through the key frame extraction algorithm built into the MVD module. After the extraction is completed, the RAW image is converted into an RGB format image through RGB (RedGreenBlue) domain processing, and the RGB image is converted into YUV format through YUV (Luminance andChrominance) domain processing for easy video transmission. Among them, MVD dynamic vector detection belongs to an image processing module in the ISP chip, which is used to detect dynamic vector information between previous and next frames. Specifically, for each video frame after preprocessing: it is used as the target video frame, and the pixel difference between the target video frame and the adjacent video frame is calculated to determine whether the target video frame is a key video frame. Specifically, the difference in pixel points between the target video frame and the adjacent previous frame (i.e., the first video frame) is calculated to obtain a first differential image. Similarly, the difference in pixel points between the target video frame and the adjacent next frame (i.e., the second video frame) is calculated to obtain a second differential image. Among them, the first differential image and the second differential image can be obtained by calculating the pixel difference of each point point by point, as shown in formulas (1) and (2):

[0052] D n (x,y)=|f n (x,y)-f n-1 (x,y)|(1)

[0053] D n+1 (x,y)=|f n+1 (x,y)-f n (x,y)|(2)

[0054] Among them, D n (x, y) is the difference between the pixel at the pixel position (x, y) of the nth frame and the n-1th frame, f n (x, y) is the grayscale value of the pixel at the pixel position (x, y) in the nth frame (i.e., the target video frame), f n-1 (x, y) is the grayscale value of the pixel at the pixel position (x, y) in the n-1th frame (i.e., the first video frame), D n+1 (x, y) is the difference between the pixel at the pixel position (x, y) of the nth frame and the n+1th frame, f n+1 (x, y) is the gray value of the pixel at the pixel position (x, y) in the n+1th frame (i.e., the second video frame). The first differential image and the second differential image are comprehensively analyzed, and the target video frame is determined to be a key video frame or a non-key video frame by comparing the change information in the two differential images.

[0055] like Figure 4 As shown in the figure, it shows the difference results between the n-1th frame, the nth frame and the n+1th frame. In the first difference image formed by the n-1th frame and the nth frame, the pendulum part is bright, indicating that the pendulum is in an obvious state of motion in these two frames. In the second difference image formed by the nth frame and the n+1th frame, the pendulum is still bright, and the black part in the above two difference images indicates a static state. It can be seen that in these three frames, the pendulum part has obvious movement changes, and the rest is relatively static, so the differential image can be used to separate the target with more obvious movement from the static target.

[0056] Specifically, in one embodiment of the present invention, the target video frame is determined to be a key video frame or a non-key video frame based on the first differential image and the second differential image:

[0057] Performing a logical AND operation on the first differential image and the second differential image to obtain a difference composite image;

[0058] Binarizing the difference composite image to obtain a binary image;

[0059] Performing connectivity analysis on the binary image to identify all connected regions in the binary image;

[0060] Based on the areas of all the connected regions, it is determined whether the target video frame is a key video frame or a non-key video frame.

[0061] like Figure 5 As shown, for the target video frame f n , and the first video frame f n-1 , the second video frame f n+1, determine whether the target video frame is a key video frame in the following way: perform pixel-by-pixel logical AND calculation on the first differential image and the second differential image of the target video frame according to formula (3):

[0062] D n ′(x,y)=D n+1 (x,y)∩D n (x,y) (3)

[0063] Among them, D n ′(x, y) is the difference synthesis value of the nth frame (i.e., the target video frame) at the pixel position (x, y). For each pixel in the first difference image and the second difference image, a logical AND operation is performed according to formula (3) to obtain a difference synthesis image, which shows the areas where changes exist in both difference images. In order to clearly separate the changed areas in the difference synthesis image and facilitate subsequent connectivity analysis, the difference synthesis image is binarized to obtain a binary image R n ′(x,y). In the binary processing, theoretically, the pixels of moving objects are set to 255, that is, white, and the pixels of stationary objects are set to 0, that is, black. In order to identify and mark all connected areas in the binary image, the connectivity analysis of the binary image is performed to obtain the connectivity image R n (x,y). Among them, connectivity analysis is to find and mark each connected area in the image so that the contour of the final moving target forms a closed loop. Specifically, the area composed of continuous white pixels is identified in the binary image and is regarded as a connected area. These connected areas represent the areas with obvious changes in the two differential images. By calculating the area of ​​these connected areas and comparing them with the preset area threshold, it is determined whether the target video frame is a key video frame or a non-key video frame.

[0064] Specifically, in one embodiment of the present invention, determining whether the target video frame is a key video frame or a non-key video frame based on the areas of all connected regions includes:

[0065] Analyze each connected region one by one:

[0066] Determine whether the area of ​​the connected region is greater than the preset area threshold:

[0067] If yes, the target video frame is the key video frame of the connected area;

[0068] Otherwise, continue to analyze the next connected area;

[0069] After all connected regions are analyzed, if the target video frame is not a key video frame of any connected region, the target video frame is regarded as a non-key video frame.

[0070] For each connected area in the binary image obtained above, analyze them one by one. For the currently analyzed connected area: by counting the number of pixels contained in the connected area, the area of ​​the connected area (i.e. the total number of pixels contained) is obtained. The area is compared with the preset area threshold to determine whether the area is greater than the area threshold: if the area is greater than the area threshold, it means that the connected area is large enough and the motion change of the area is obvious, so the target video frame is used as the key video frame and its frame number is recorded. Otherwise, it is considered that the motion change of the area is not obvious, and the next connected area is analyzed, and the above process is repeated. If all the connected areas in the binary image obtained above are analyzed, and the area of ​​no area exceeds the area threshold, the target video frame will be used as a non-key video frame.

[0071] S203: Downsample the non-key video frame.

[0072] After the current video stream to be processed is processed by the ISP module, key video frames and non-key video frames have been distinguished. At this time, the wearable device will downsample the non-key video frames. Specifically, for the video stream, by downsampling frame by frame and identifying the frame number of the key video frame, the key video frame is skipped and not downsampled. For each non-key video frame in the current video stream, the maximum pooling or average pooling method can be used for frame-by-frame downsampling. Figure 6 As shown, maximum pooling refers to selecting the maximum pixel value in each area, and average pooling refers to selecting the average value of pixels in each area. By downsampling, the resolution of non-critical video frames is effectively reduced, thereby reducing the amount of data. It is understandable that in order to ensure that the quality of video transmission is not affected, the present invention also keeps the key video frames unchanged, and only downsamples the non-critical video frames. In this way, the key content in the video is retained while reducing the amount of data and improving the transmission rate.

[0073] S204: Send the non-key video frame after downsampling and the corresponding key video frame to a smart terminal.

[0074] The wearable device packages the video stream composed of each non-key video frame after downsampling and the corresponding key video frame, and sends it to the smart terminal through wireless transmission according to the corresponding wireless transmission protocol.

[0075] As shown in FIG7 , the present invention also provides a video calling method, which is applied to a smart terminal, and the method comprises the following steps:

[0076] S701: Receive a key video frame and a corresponding non-key video frame sent by a wearable device.

[0077] After the smart terminal establishes a data connection with the wearable device, it receives each key video frame and the corresponding non-key video frame processed by the wearable device through a wireless transmission method such as cellular or WiFi. It is understandable that the smart terminal includes but is not limited to smart phones, tablets, laptops, desktop computers, smart TVs, etc., as long as it can realize video calls with wearable devices, and is not limited here.

[0078] S702: The key video frame and the non-key video frame are input into a lightweight super-resolution model, and super-resolution restoration is performed on the non-key video frame based on the key video frame.

[0079] A lightweight super-resolution model is deployed on the smart terminal. After receiving the downsampled video stream sent by the wearable device, a lightweight super-resolution algorithm is used to extract the feature information of the key video frames and fuse it with the low-resolution features of the non-key video frames to restore the resolution of the original image.

[0080] Specifically, in one embodiment of the present invention, the step of inputting the key video frame and the non-key video frame into a lightweight super-resolution model, and performing super-resolution restoration on the non-key video frame according to the key video frame includes:

[0081] Inputting the key video frame into the first feature extraction module of the lightweight super-resolution model, performing feature extraction on the key frame video, and obtaining a key frame feature map;

[0082] Inputting the non-key video frame into the second feature extraction module of the lightweight super-resolution model, performing feature extraction on the non-key video frame to obtain a non-key frame feature map;

[0083] Inputting the key frame feature map and the non-key frame feature map into the feature fusion module of the lightweight super-resolution model, performing feature fusion on the key frame feature map and the non-key frame feature map to obtain a fused feature map;

[0084] The fused feature map is input into the basic residual module group of the lightweight super-resolution model, feature processing is performed through each basic residual module to obtain a residual feature map, and all the residual feature maps are subjected to multi-level residual mapping to obtain an enhanced feature map; wherein the basic residual module group includes a plurality of cascaded basic residual modules;

[0085] The enhanced feature map is input into the up-sampling module of the lightweight super-resolution model, and the enhanced feature map is up-sampled to obtain a non-critical video frame after super-resolution restoration.

[0086] The present invention uses a lightweight super-resolution algorithm to restore the original image resolution. An example of a lightweight super-resolution algorithm based on deep learning is provided, namely a real-time super-resolution algorithm based on key frame constraints. The super-resolution network structure is simple and meets the requirements of real-time video. Secondly, through the constraints of key frames, the lightweight super-resolution algorithm has higher robustness and can highly restore the original image.

[0087] Specifically, the lightweight super-resolution model described in the present invention mainly includes a basic network architecture and a basic residual block. For the basic network architecture, the design architecture of the lightweight super-resolution model is as follows: Figure 8 As shown. The key video frame is input into the first feature extraction module of the lightweight super-resolution model (composed of one or more convolutional layers, such as a 5*5 convolutional layer), and the image features in the key video frame are extracted by convolution operation to obtain a key frame feature map. In addition, a non-key video frame corresponding to the key video frame is input into the second feature extraction module of the model (composed of one or more convolutional layers, such as a 3*3 convolutional layer), and the image features in the input non-key video frame are extracted by convolution operation to obtain a non-key frame feature map. The two feature maps are input into the feature fusion module for feature fusion. By fusing the key frame features containing more detail features with the non-key features containing less detail information, the relevant details of the non-key frame can be effectively restored through the relevant features of the key frame, so that the restored image is close to the original image. The fused feature map is input into the basic residual module group. Since the basic residual module group includes at least two cascaded basic residual modules, each basic residual module further extracts the features of the input feature map on the basis of retaining the input feature map information through operations such as feature extraction and residual connection. Through the processing of these basic residual modules, a gradually refined and deeper residual feature map can be obtained. The residual feature maps obtained by all basic residual modules are subjected to multi-level residual mapping, and all residual feature maps are connected together to form an enhanced feature map by accumulation or fusion. The enhanced feature map is upsampled through a convolution layer and an upsampling module, so that the resolution of the enhanced feature map is increased to the target resolution, and non-critical video frames after super-resolution restoration are generated, thereby obtaining a restored high-resolution image, which is closer to the image before downsampling. This basic network architecture ensures the lightweight of the network and keeps the network parameters within a sufficient range, so that smart terminals such as mobile phones can perform real-time processing on downsampled video streams.

[0088] In one embodiment of the present invention, the step of performing feature processing on each basic residual module to obtain a residual feature map includes:

[0089] Inputting the fused feature map / residual feature map into the feature extraction unit of the corresponding basic residual module for feature extraction to obtain a deep feature map;

[0090] The depth feature map and the fused feature map / residual feature map are input into the residual connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

[0091] The basic residual module of the present invention is as follows Fig. 9 As shown, for each basic residual module, the processing process of the module on the input fused feature map / residual feature map is as follows: the feature map output by the previous module is used as input data and passed to the basic residual module. Specifically, if the basic residual module is the first module in the basic residual module group, the input data is the fused feature map. Conversely, if the basic residual module is not the first module in the basic residual module group, the input data is the residual feature map generated by the previous basic residual module. The input fused feature map / residual feature map passes through multiple convolution layers (such as conv3, conv1) with different convolution kernel sizes in turn, and finally generates a deep feature map by extracting detail information in the image layer by layer. The processed deep feature map is residually connected with the input feature map to obtain the residual feature map as the output of the current basic residual module. It should be noted that the specialLayer layer in the figure can be adaptively designed by those skilled in the art based on actual needs, for example, it can be a depth separable convolution, a gradient guided layer, or an attention mechanism layer, etc., which is not limited here.

[0092] The present invention adjusts the skip links and allows each convolutional basic block to fully exert its function without redundancy, which is the key to the image quality restored by the final high-resolution algorithm. The basic residual module in the lightweight super-resolution model is optimized in the following ways:

[0093] In another embodiment of the present invention, a basic residual module with channel segmentation having a pixel attention mechanism is provided, wherein feature processing is performed on each basic residual module to obtain a residual feature map. The method comprises:

[0094] Inputting the fused feature map / residual feature map into the first feature extraction unit of the corresponding basic residual module to extract the first feature, and enhancing the extracted first feature based on the pixel attention mechanism to obtain an enhanced feature map;

[0095] Input the fused feature map / residual feature map into the second feature extraction unit of the corresponding basic residual module to extract the second feature, and process the second feature on the basis of retaining the features of the fused feature map / residual feature map to obtain a basic feature map;

[0096] Inputting the enhanced feature map and the basic feature map into the corresponding splicing unit of the basic residual module for feature splicing to obtain a comprehensive feature map;

[0097] The comprehensive feature map and the fused feature map / residual feature map are input into the connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

[0098] like Fig.10 As shown, the present invention also provides another basic residual module of the architecture, which draws on the idea of ​​channel segmentation in the self-calibration convolution and includes two branches, upper and lower. Specifically, the fused feature map or residual feature map is input into the upper branch of the module, and the first feature in the input feature map is extracted through a 1*1 convolution layer, and the extracted first feature is processed through two 3*3 convolution layers, wherein the first convolution layer further strengthens these features through the pixel attention mechanism, so that the feature map can be enhanced in the key area, and after passing through the last 3*3 convolution layer, an enhanced feature map is generated. At the same time, the fused feature map or residual feature map is also input into the lower branch of the module, and the second feature is extracted from the input feature map through a 1*1 convolution layer. And through a single 3*3 convolution layer, while retaining the basic feature information in the input feature map, the extracted second feature is processed to generate a basic feature map, and the original feature information is retained in the basic feature map. The outputs of the upper and lower branches are connected, and the obtained comprehensive feature map contains the enhanced key feature information and the original feature information. After the comprehensive feature map passes through a 1*1 convolution layer, a residual connection is performed with the input fused feature map or residual feature map, so that the residual feature map finally generated by the basic residual module retains the original input features while also integrating the enhanced new features. It should be noted that the specialLayer layer in the figure can be adaptively designed by technicians in this field based on actual needs, for example, it can be a depth-separable convolution, a gradient-guided layer, or an attention mechanism layer, etc., which is not limited here.

[0099] In super-resolution algorithms, channel segmentation can effectively improve the efficiency of feature information, mainly in the following aspects: (1) Channel segmentation can improve feature expression ability. Through channel segmentation, the algorithm can learn richer and more differentiated features on different channels, thereby improving the feature expression ability. (2) Secondly, channel segmentation can also enhance the generalization ability of the model: different channels can focus on capturing different features of the image, so that even in the case of limited data sets, the model's generalization ability for new images can be improved. (3) Channel segmentation can also often reduce computing resource consumption, allowing the network to perform calculations only on necessary channels when processing high-resolution images, thereby reducing computing resource consumption.

[0100] In yet another embodiment of the present invention, a basic residual module for channel segmentation is provided, wherein feature processing is performed on each basic residual module to obtain a residual feature map, including:

[0101] Inputting the fused feature map / residual feature map into the first feature extraction unit of the corresponding basic residual module to extract the first feature to obtain an enhanced feature map;

[0102] Input the fused feature map / residual feature map into the second feature extraction unit of the corresponding basic residual module to extract the second feature, and process the second feature on the basis of retaining the features of the fused feature map / residual feature map to obtain a basic feature map;

[0103] Inputting the enhanced feature map and the basic feature map into the corresponding splicing unit of the basic residual module for feature splicing to obtain a comprehensive feature map;

[0104] The comprehensive feature map and the fused feature map / residual feature map are input into the connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

[0105] like Fig.11 As shown, this embodiment uses the basic residual block of channel segmentation, and draws on the architecture of the spatial attention module of the self-calibration network. Its processing process is roughly the same as the basic residual module of channel segmentation with the pixel attention mechanism mentioned above, which will not be described in detail here. It can be understood that compared with the previous embodiment, this embodiment is relatively insufficient in feature extraction of key areas of the image because there is no pixel attention mechanism. Since the use of the pixel attention mechanism will lead to local enhancement, it will cause the model to overfit on specific data, so this embodiment is more prominent in the generalization performance of the model. It should be noted that the specialLayer layer in the figure can be adaptively designed by technicians in this field based on actual needs. For example, it can be a depth separable convolution, a gradient guided layer, or an attention mechanism layer, etc., which is not limited here.

[0106] In yet another embodiment of the present invention, a basic residual module composed of shallow residual blocks is provided. The feature processing is performed on each basic residual module to obtain a residual feature map, including:

[0107] The fused feature map / residual feature map is input into the feature enhancement unit of the corresponding basic residual module, features are extracted layer by layer through multiple convolutional layers, and residual connection is performed with the input features of the corresponding convolutional layer after each feature extraction to obtain an enhanced feature map;

[0108] The fused feature map / residual feature map and the enhanced feature map are input into the connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

[0109] like Fig.12As shown, in this embodiment, the input fused feature map or residual feature map is processed by using multiple cascaded shallow residual blocks (SRBs) to extract and enhance the features of the input image. Specifically, for each shallow residual block: Fig.13 As shown, a 3*3 convolution layer is used to extract the image features of the shallow residual block input, and the extracted features are nonlinearly processed by the ReLU activation function. By performing a residual connection between the input image features and the features after nonlinear processing, the obtained feature map contains both the features of the original input and the feature expression enhanced by the convolution operation. The output of the last shallow residual block is used as the enhanced feature map, and after passing through a 1*1 convolution layer, it is residually connected with the fused feature map / residual feature map input by the current basic residual module to obtain a residual feature map. Since the shallow residual block has a simple structure and a small number of parameters, the model has a faster processing rate during training and reasoning, greatly reducing memory resources. It should be noted that the specialLayer layer in the figure can be adaptively designed by technicians in this field based on actual needs, for example, it can be a depth-separable convolution, a gradient-guided layer, or an attention mechanism layer, etc., which is not limited here.

[0110] In another embodiment of the present invention, a basic residual module composed of channel segmentation and shallow residual blocks is provided. The feature processing is performed on each basic residual module to obtain a residual feature map, including:

[0111] Input the fused feature map / residual feature map to the first feature processing unit of the corresponding basic residual module, extract features layer by layer through multiple convolutional layers, and perform residual connection with the input features of the corresponding convolutional layer after each feature extraction to obtain an enhanced feature map;

[0112] Input the fused feature map / residual feature map into the second feature unit of the corresponding basic residual module, extract the features of the fused feature map / residual feature map, and process the extracted features on the basis of retaining the features of the fused feature map / residual feature map to obtain a basic feature map;

[0113] Inputting the enhanced feature map and the basic feature map into the corresponding splicing unit of the basic residual module for feature splicing to obtain a spliced ​​feature map;

[0114] The concatenated feature map and the fused feature map / residual feature map are input into the connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

[0115] like Fig.14As shown in the figure, the basic residual module is divided into two upper and lower branches. The fused feature map / residual feature map is input into the upper branch. Local features are extracted and enhanced step by step through multiple shallow residual blocks, and an enhanced feature map is generated in the last shallow residual block. After the feature map of the lower branch is subjected to a 1*1 convolution layer to extract basic features, it is further processed by a 3*3 convolution layer to retain the original information of the input fused feature map / residual feature map while extracting relevant features, thereby obtaining a basic feature map. After splicing these two feature maps, a residual connection is performed with the input fused feature map / residual feature map, and finally the residual feature map output by the basic residual module is obtained. By combining the shallow residual block with the channel segmentation idea, multi-level feature extraction can be effectively realized, and the generalization ability and adaptability of the model can be improved. It should be noted that the specialLayer layer in the figure can be adaptively designed by technicians in this field based on actual needs, for example, it can be a depth separable convolution, a gradient guided layer, or an attention mechanism layer, etc., which is not limited here.

[0116] S203: Arrange the key video frames and the restored non-key video frames in sequence and present them as a target video.

[0117] After the above processing, the smart terminal arranges each key video frame and the corresponding restored non-key video frames in order according to the frame number or timestamp, so as to form a target video stream to be displayed on the smart terminal. Fig.15 As shown in the figure, it can be seen that after the downsampling, the resolution of the image is greatly reduced, resulting in the loss of relevant details of the bird's eye and feathers in the image, and the image is blurred. Compared with the downsampled image, the image after super-resolution restoration by the method of the present invention has a clearer texture and can be closer to the original image.

[0118] In one embodiment of the present invention, a video call system is also provided, and the video call system corresponds to the video call method in the above embodiment. Fig.16As shown, the video call system includes a wearable device end and a smart terminal that are connected to each other in communication, wherein the wearable device end includes: a video frame extraction module 1601, a video frame screening module 1602, a downsampling module 1603 and a first communication module 1604. The smart terminal includes: a second communication module 1605, a super-resolution restoration module 1606 and a presentation module 1607. The functional modules are described in detail as follows: the video frame extraction module 1601 is used to obtain the video stream of the video call and extract all video frames from the video stream; the video frame screening module 1602 is used to determine the key video frame and the corresponding non-key video frame from all the video frames based on the dynamic vector detection algorithm. The downsampling module 1603 is used to downsample the non-key video frame. The first communication module 1604 is used to send the non-key video frame and the corresponding key video frame after downsampling to the smart terminal. The second communication module 1605 is used to receive the key video frame and the corresponding non-key video frame sent by the wearable device end. The super-resolution restoration module 1606 is used to input the key video frame and the non-key video frame into the lightweight super-resolution model, and perform super-resolution restoration on the non-key video frame according to the key video frame. The presentation module 1607 is used to arrange the key video frame and the restored non-key video frame in sequence and present them as the target video.

[0119] For the specific definition of the video call system, please refer to the definition of the video call method above, which will not be repeated here. Each module in the above video call system can be implemented in whole or in part by software, hardware and their combination. The above modules can be embedded in or independent of the processor in the computer device in hardware format, or stored in the memory of the computer device in software format, so that the processor can call the corresponding operations of each of the above modules.

[0120] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0121] In summary, the present invention discloses a video call method and system, and the present invention proposes a new technical framework, which makes full use of the hardware performance of the smart terminal side when the smart terminal and the wearable device make a video call, thereby reducing the power consumption of the wearable device side such as the smart watch. The present invention also proposes a new super-resolution algorithm, that is, a real-time super-resolution algorithm based on key frame constraints, which uses the MVD module in the ISP to detect the key frames in the video stream, and uses the key frames to constrain the super-resolution algorithm, so that the super-resolution algorithm can restore the original image more accurately. The present invention proposes a real-time super-resolution algorithm processing framework, which fully guarantees the real-time performance of the smart terminal and the wearable device during the video call through a lightweight network model such as a basic residual block. With the solution of the present invention, since the image resolution after downsampling becomes smaller, the effective data that needs to be transmitted during the video call becomes less, so the traffic consumption during the video call is greatly reduced, so that the performance consumption of the codec during the video transmission is reduced, and the performance of the wearable device such as the smart watch during the video call is improved as a whole. In addition, this method reduces the power consumption requirements of the video during the wireless transmission process, thereby increasing the battery life of the wearable device. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0122] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A video calling method, characterized in that: Applied to a wearable device, the method comprises: Obtaining a video stream of a video call, and extracting all video frames from the video stream; Based on a dynamic vector detection algorithm, determining key video frames and corresponding non-key video frames from all the video frames; Performing downsampling processing on the non-key video frames; The non-key video frame after downsampling processing and the corresponding key video frame are sent to the smart terminal.

2. The video calling method according to claim 1, characterized in that: The method of determining the key video frames and the corresponding non-key video frames from all the video frames based on the dynamic vector detection algorithm includes: Preprocessing all the video frames; Analyze all the preprocessed video frames, and analyze each video frame as a target video frame: Obtaining a first differential image between the target video frame and a first video frame adjacent thereto based on the difference in pixel points between the target video frame and the first video frame adjacent thereto; wherein the first video frame is located before the target video frame; Based on the difference of the pixel points between the target video frame and a second video frame adjacent thereto, a second differential image between the two is obtained; wherein the second video frame is located after the target video frame; Based on the first differential image and the second differential image, it is determined whether the target video frame is a key video frame or a non-key video frame.

3. The video calling method according to claim 2, characterized in that: The step of determining whether the target video frame is a key video frame or a non-key video frame based on the first differential image and the second differential image: Performing a logical AND operation on the first differential image and the second differential image to obtain a difference composite image; Binarizing the difference composite image to obtain a binary image; Performing connectivity analysis on the binary image to identify all connected regions in the binary image; Based on the areas of all the connected regions, it is determined whether the target video frame is a key video frame or a non-key video frame.

4. The video calling method according to claim 3, characterized in that: The determining, based on the areas of all the connected regions, whether the target video frame is a key video frame or a non-key video frame comprises: Analyze each connected region one by one: Determine whether the area of ​​the connected region is greater than the preset area threshold: If yes, the target video frame is the key video frame of the connected area; Otherwise, continue to analyze the next connected area; After all connected regions are analyzed, if the target video frame is not a key video frame of any connected region, the target video frame is regarded as a non-key video frame.

5. A video calling method, characterized in that: Applied to a smart terminal, the method comprises: Receive key video frames and corresponding non-key video frames sent by the wearable device; Inputting the key video frame and the non-key video frame into a lightweight super-resolution model, and performing super-resolution restoration on the non-key video frame according to the key video frame; The key video frames and the restored non-key video frames are arranged in sequence and presented as a target video.

6. The video calling method according to claim 5, characterized in that: The step of inputting the key video frame and the non-key video frame into a lightweight super-resolution model and performing super-resolution restoration on the non-key video frame according to the key video frame comprises: Inputting the key video frame into the first feature extraction module of the lightweight super-resolution model, performing feature extraction on the key frame video, and obtaining a key frame feature map; Inputting the non-key video frame into the second feature extraction module of the lightweight super-resolution model, performing feature extraction on the non-key video frame to obtain a non-key frame feature map; Inputting the key frame feature map and the non-key frame feature map into the feature fusion module of the lightweight super-resolution model, performing feature fusion on the key frame feature map and the non-key frame feature map to obtain a fused feature map; The fused feature map is input into the basic residual module group of the lightweight super-resolution model, feature processing is performed through each basic residual module to obtain a residual feature map, and all the residual feature maps are subjected to multi-level residual mapping to obtain an enhanced feature map; wherein the basic residual module group includes a plurality of cascaded basic residual modules; The enhanced feature map is input into the up-sampling module of the lightweight super-resolution model, and the enhanced feature map is up-sampled to obtain a non-critical video frame after super-resolution restoration.

7. The video calling method according to claim 6, characterized in that: The feature processing is performed through each basic residual module to obtain a residual feature map, including: Inputting the fused feature map / residual feature map into the feature extraction unit of the corresponding basic residual module for feature extraction to obtain a deep feature map; The depth feature map and the fused feature map / residual feature map are input into the residual connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

8. The video calling method according to claim 6, characterized in that: The feature processing is performed through each basic residual module to obtain a residual feature map, including: Inputting the fused feature map / residual feature map into the first feature extraction unit of the corresponding basic residual module to extract the first feature, and enhancing the extracted first feature based on the pixel attention mechanism to obtain an enhanced feature map; Input the fused feature map / residual feature map to the second feature extraction unit of the corresponding basic residual module to extract the second feature, and process the second feature on the basis of retaining the features of the fused feature map / residual feature map to obtain a basic feature map; Inputting the enhanced feature map and the basic feature map into the corresponding splicing unit of the basic residual module for feature splicing to obtain a comprehensive feature map; The comprehensive feature map and the fused feature map / residual feature map are input into the connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

9. The video calling method according to claim 6, characterized in that: The feature processing is performed through each basic residual module to obtain a residual feature map, including: The fused feature map / residual feature map is input into the feature enhancement unit of the corresponding basic residual module, features are extracted layer by layer through multiple convolutional layers, and residual connection is performed with the input features of the corresponding convolutional layer after each feature extraction to obtain an enhanced feature map; The fused feature map / residual feature map and the enhanced feature map are input into the connection unit of the corresponding basic residual module for residual connection to obtain a residual feature map.

10. A video call system, characterized in that: The system includes a wearable device and a smart terminal that are communicatively connected to each other: The wearable device terminal includes: A video frame extraction module, used to obtain a video stream of a video call and extract all video frames from the video stream; A video frame screening module, used for determining key video frames and corresponding non-key video frames from all the video frames based on a dynamic vector detection algorithm; A downsampling module, used for downsampling the non-key video frames; A first communication module, used for sending the non-key video frame after downsampling processing and the corresponding key video frame to a smart terminal; The intelligent terminal comprises: A second communication module is used to receive key video frames and corresponding non-key video frames sent by the wearable device; A super-resolution restoration module, used for inputting the key video frame and the non-key video frame into a lightweight super-resolution model, and performing super-resolution restoration on the non-key video frame according to the key video frame; The presentation module is used to arrange the key video frames and the restored non-key video frames in sequence and present them as a target video.

Images