Video communication method and device, electronic equipment and readable storage medium

Abstract

The invention provides a video communication method and device, electronic equipment and a readable storage medium. The method comprises the following steps: selecting a key frame from a video frame sequence to be transmitted according to characteristics and channel conditions of a video to obtain a key frame sequence; performing source-channel joint coding on the key frame sequence; and sending the coded key frame sequence to a receiving end device. In the application, a sending end device selects a key frame based on characteristics of a video and channel conditions, encodes the key frame and sends the encoded key frame to a receiving end device. The self-adaptive key frame selection method can ensure the communication reliability. Only the coded key frame is transmitted during video communication, so that the processing delay of the video frame is reduced, the channel bandwidth can be saved, and the transmission efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a video communication method, device, electronic device and readable storage medium. Background Art

[0002] As human society becomes increasingly dependent on digital and networked life, the demand for the Internet is growing explosively. For special scenarios served by future 6G, such as virtual reality (VR) and telemedicine, the demand for real-time video communication is particularly urgent. In these scenarios, wireless video communication not only needs to provide high-definition, low-latency video transmission, but also needs to ensure high reliability and effectiveness of the transmission process.

[0003] However, the traditional source-channel separation coding scheme cannot meet the low latency requirements of 6G, and a cliff effect will occur when the channel environment is poor. In recent years, the rapid development of AI technology and machine learning (ML) has brought revolutionary changes to the field of wireless communication technology, providing new ideas and possibilities for the research and development of 6G communication systems. The deep JSCC method closely combines source coding and channel coding through an end-to-end deep learning model, achieving joint optimization of source and channel characteristics.

[0004] At present, the wireless video semantic communication methods based on deep JSCC use deep neural networks to directly map video signals to channel symbols for compression to overcome the cliff effect. However, most of these methods use complex transmitter designs, focus on improving the quality of video reconstruction, and ignore the problem of long processing delay. When processing video data streams, efficiency and real-time performance are crucial to user experience and the overall performance of the communication system. Therefore, how to effectively reduce processing delay while ensuring the quality of video reconstruction has become a major challenge that needs to be solved in the current field of wireless video semantic communication. Summary of the invention

[0005] The purpose of the present application is to provide a video communication method, device, electronic device and readable storage medium, which solve the problem of long delay in the existing video communication process.

[0006] To achieve the above object, an embodiment of the present application provides a video communication method, which is applied to a transmitting device, including:

[0007] According to the characteristics of the video and the channel conditions, key frames are selected from the video frame sequence to be transmitted to obtain a key frame sequence;

[0008] Performing source-channel joint coding on the key frame sequence;

[0009] The encoded key frame sequence is sent to a receiving device.

[0010] Optionally, according to characteristics of the video and channel conditions, key frames are selected from the video frame sequence to be transmitted to obtain a key frame sequence, including:

[0011] Determine the transmission rate based on the characteristics of the video;

[0012] Determining the number and positions of key frames according to the transmission rate and channel conditions;

[0013] Key frames are selected from the video frame sequence according to the quantity and position to obtain a key frame sequence.

[0014] Optionally, the characteristics of the video include: frame rate and / or resolution of the video;

[0015] Determining the transmission rate according to the characteristics of the video includes:

[0016] The transmission rate is calculated based on the video's frame rate, resolution, inter-frame compression multiples of selected key frames, and intra-frame compression multiples of the encoding.

[0017] Optionally, the channel condition includes: channel bandwidth and / or signal-to-noise ratio;

[0018] Determining the number and position of key frames according to the transmission rate and channel conditions includes:

[0019] Determine a minimum inter-frame compression multiple for selecting a key frame according to the transmission rate and the channel capacity, wherein the transmission rate is less than or equal to the channel capacity, and the channel capacity is calculated according to the channel bandwidth and the signal-to-noise ratio;

[0020] Calculating the number of key frames according to the minimum inter-frame compression multiple;

[0021] The position of the key frame in the video frame sequence is determined according to the number of the key frames and a preset sampling method.

[0022] Optionally, performing source-channel joint coding on the key frame sequence includes:

[0023] Performing semantic extraction on the key frame sequence to obtain a semantic feature map of the key frame sequence;

[0024] Perform source-channel joint coding on the semantic feature map to obtain a coding sequence.

[0025] Optionally, before sending the encoded key frame sequence to the receiving end device, the method further includes:

[0026] quantifying the coding sequence;

[0027] The step of sending the encoded key frame sequence to a receiving device comprises:

[0028] Send the quantized code sequence to the receiving device.

[0029] Optionally, before selecting key frames from the video frame sequence to be transmitted according to the characteristics of the video frame sequence to be transmitted and the channel conditions to obtain the key frame sequence, the method further includes:

[0030] Perform video capture, determine the frame rate and resolution of the video, and obtain the video frame sequence.

[0031] To achieve the above object, an embodiment of the present application provides a video communication method, which is applied to a receiving device, including:

[0032] receiving a key frame sequence;

[0033] Performing source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence;

[0034] Performing video frame insertion on the reconstructed key frame sequence according to the number and positions of the key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence;

[0035] The number and position of the key frames are related to the characteristics of the video and the channel conditions.

[0036] Optionally, performing video frame interpolation on the reconstructed key frame sequence according to the number and positions of key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence includes:

[0037] Determining the position and number of video interpolation frames according to the position and number of key frames in the reconstructed key frame sequence;

[0038] Video frame insertion is performed on the reconstructed key frame sequence according to the position and number of the video frame insertion to obtain a reconstructed video frame sequence.

[0039] Optionally, the characteristics of the video include a frame rate and / or a resolution of the video;

[0040] The channel conditions include: channel bandwidth and / or signal-to-noise ratio;

[0041] The number and position of the key frames are determined according to the transmission rate and channel conditions; the transmission rate is calculated according to the frame rate and resolution of the video.

[0042] Optionally, performing source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence includes:

[0043] Performing source-channel joint decoding on the key frame sequence to obtain a reconstructed semantic feature map;

[0044] Perform semantic restoration on the reconstructed semantic feature map to obtain a reconstructed key frame sequence.

[0045] Optionally, the method further includes:

[0046] The reconstructed video frame sequence is played at the frame rate of the video.

[0047] To achieve the above-mentioned object, an embodiment of the present application provides a video communication device, which is applied to a transmitting end device, including:

[0048] A key frame selection module is used to select key frames from the video frame sequence to be transmitted according to the characteristics of the video and the channel conditions to obtain a key frame sequence;

[0049] A coding module, used for performing source-channel joint coding on the key frame sequence;

[0050] The wireless transmission module is used to send the encoded key frame sequence to a receiving device.

[0051] To achieve the above object, an embodiment of the present application provides a video communication device, which is applied to a receiving device, including:

[0052] A receiving module, used for receiving a key frame sequence;

[0053] A decoding module, used for performing source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence;

[0054] A video frame insertion module, configured to perform video frame insertion on the reconstructed key frame sequence according to the number and positions of the key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence;

[0055] The number and position of the key frames are related to the characteristics of the video and the channel conditions.

[0056] To achieve the above-mentioned purpose, an embodiment of the present application provides an electronic device, including a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the steps of the above-mentioned video communication method are implemented.

[0057] To achieve the above-mentioned purpose, an embodiment of the present application provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above-mentioned video communication method are implemented.

[0058] To achieve the above-mentioned purpose, an embodiment of the present application provides a computer program product on which a program or instruction is stored, including computer instructions, which implement the steps of the above-mentioned video communication method when executed by a processor.

[0059] The beneficial effects of the above technical solution of the present application are as follows:

[0060] In the embodiment of the present application, the transmitting device selects key frames based on the characteristics of the video itself and the channel conditions, and encodes the key frames and sends them to the receiving device. The method of adaptively selecting key frames can ensure the reliability of communication. During video communication, only the encoded key frames are transmitted, which reduces the processing delay of the video frames, saves channel bandwidth, and improves transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is one of the flowcharts of the video communication method according to an embodiment of the present application;

[0062] Figure 2 This is a second flowchart of the video communication method according to an embodiment of the present application;

[0063] Figure 3 A convolutional network for semantic extraction in an embodiment of the present application;

[0064] Figure 4 A convolutional network for source-channel joint coding in an embodiment of the present application;

[0065] Figure 5 A convolutional network for source-channel joint decoding in an embodiment of the present application;

[0066] Figure 6 A convolutional network for semantic restoration in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of video frame insertion according to an embodiment of the present application;

[0068] Figure 8 This is a schematic diagram of the effect of reconstruction quality and delay of the video communication method of the embodiment of the present application and the traditional video coding transmission solution;

[0069] Fig. 9 The second schematic diagram is a diagram showing the effect of reconstruction quality and delay of the video communication method according to the embodiment of the present application and the traditional video coding transmission scheme;

[0070] Fig.10 This is a third flowchart of the video communication method according to an embodiment of the present application;

[0071] Fig.11 This is one of the structural schematic diagrams of the video communication device according to an embodiment of the present application;

[0072] Fig.12 This is a second structural diagram of the video communication device according to an embodiment of the present application;

[0073] Fig.13 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the technical problems, technical solutions and advantages to be solved by the present application clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0075] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0076] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0077] Additionally, the terms "system" and "network" are often used interchangeably herein.

[0078] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0079] like Figure 1 As shown, an embodiment of the present application provides a video communication method, which is applied to a transmitting end device, including:

[0080] Step 11: Select key frames from the video frame sequence to be transmitted according to the characteristics of the video and the channel conditions to obtain a key frame sequence;

[0081] Step 12: performing source-channel joint coding on the key frame sequence;

[0082] Step 13: Send the encoded key frame sequence to a receiving device.

[0083] In this embodiment, the transmitting device selects some key frames from the video frame sequence to be transmitted, processes them, and then sends them to the receiving device. The selected key frames are related to the characteristics of the video and the channel conditions. On the one hand, the channel conditions affect the reliability of communication. For example, when the channel conditions are good and the channel capacity is large, the code length can be appropriately increased to increase the transmission rate, but it is always necessary to ensure that the transmission rate is lower than the channel capacity; when the channel conditions are bad, the channel capacity decreases accordingly, and the code length should be reduced to reduce the transmission rate to ensure the reliability of communication. On the other hand, the characteristics of the video source itself will also affect the transmission rate, that is, it will also affect the reliability of communication.

[0084] In the embodiment of the present application, the transmitting device selects key frames based on the characteristics of the video itself and the channel conditions, and encodes the key frames and sends them to the receiving device. The method of adaptively selecting key frames can ensure the reliability of communication. During video communication, only the encoded key frames are transmitted, which reduces the processing delay of the video frames, saves channel bandwidth, and improves transmission efficiency.

[0085] As an optional embodiment, according to the characteristics of the video and the channel conditions, a key frame is selected from the video frame sequence to be transmitted to obtain a key frame sequence, including:

[0086] Determine the transmission rate based on the characteristics of the video;

[0087] Determining the number and positions of key frames according to the transmission rate and channel conditions;

[0088] Key frames are selected from the video frame sequence according to the quantity and position to obtain a key frame sequence.

[0089] Optionally, the characteristics of the video include the frame rate and / or resolution of the video. Since the frame rate, resolution, etc. of the video source itself will also affect the transmission rate, in this embodiment, the transmitting end device determines the transmission rate according to the characteristics of the video, which can ensure the reliability of communication. Optionally, the channel conditions include channel bandwidth and / or signal-to-noise ratio. The channel conditions will affect the channel capacity. The transmission rate needs to be less than the channel capacity. The number and position of the key frames to be selected are determined based on the transmission rate and channel capacity conditions, and then the key frame sequence is obtained. Adaptive transmission rate determination and key frame selection are achieved.

[0090] Each time the transmitting device transmits a video, it can adaptively select the required key frames according to the characteristics of the video and the channel conditions, and transmit the key frames to the receiving end, thereby reducing the processing flow of the video frames and saving transmission resources.

[0091] Optionally, the characteristics of the video include: frame rate and / or resolution of the video;

[0092] The determining of the transmission rate according to the characteristics of the video includes: calculating the transmission rate according to the frame rate and resolution of the video, the inter-frame compression multiple of the selected key frames, and the intra-frame compression multiple of the encoded frames.

[0093] Specifically, the calculation formula of the transmission rate R is as follows: R = r × w × h ÷ ρ ÷ θ. Among them, r is the frame rate of the video, w × h is the resolution of the video, ρ is the inter-frame compression factor of the key frame selector, and θ is the intra-frame compression factor of the source channel joint encoder. The inter-frame compression factor can be used to indicate how many frames a key frame is selected.

[0094] Optionally, the channel condition includes: channel bandwidth and / or signal-to-noise ratio;

[0095] Determining the number and position of key frames according to the transmission rate and channel conditions includes:

[0096] Determine a minimum inter-frame compression multiple for selecting a key frame according to the transmission rate and the channel capacity, wherein the transmission rate is less than or equal to the channel capacity, and the channel capacity is calculated according to the channel bandwidth and the signal-to-noise ratio;

[0097] Calculating the number of key frames according to the minimum inter-frame compression multiple;

[0098] The position of the key frame in the video frame sequence is determined according to the number of the key frames and a preset sampling method.

[0099] In this embodiment, since the transmission rate R is less than or equal to the channel capacity C, according to the calculation formula of the transmission rate R, the minimum inter-frame compression multiple ρ can be solved: min , the total number of video frames and the minimum inter-frame compression multiple length ρ are known min , the number of key frame selections k can be calculated.

[0100] Among them, the channel capacity Where W is the channel bandwidth, is the signal-to-noise ratio, and its conversion formula is When the transmission rate R>C, reliable transmission cannot be achieved, so it is necessary to ensure that the code rate R is lower than the channel capacity C.

[0101] Optionally, the device for determining the channel condition may be a spectrum analysis device at the transmitting end, which is used to obtain the channel bandwidth, signal-to-noise ratio, etc. For example, a spectrum analyzer is used for measurement, which can receive an input signal and convert it into a spectrum diagram or a spectrum density diagram for display.

[0102] As an optional embodiment, performing source-channel joint coding on the key frame sequence includes:

[0103] Performing semantic extraction on the key frame sequence to obtain a semantic feature map of the key frame sequence;

[0104] Perform source-channel joint coding on the semantic feature map to obtain a coding sequence.

[0105] In this embodiment, the encoding device can be a semantic encoding device at the transmitting end, which is used to convert the key frame sequence to be transmitted into an electrical signal suitable for transmission, adaptively adjust the transmission rate according to the relevant parameters of the video determined by the sensor device and the channel bandwidth and signal-to-noise ratio measured by the spectrum analysis device, and after determining the number of key frames to be selected, select the key frame sequence to be transmitted, encode and quantize the key frame sequence, reduce the bandwidth of the transmitted data, and improve the transmission efficiency.

[0106] Optionally, before sending the encoded key frame sequence to the receiving end device, the method further includes:

[0107] quantifying the coding sequence;

[0108] The step of sending the encoded key frame sequence to a receiving device comprises:

[0109] Send the quantized code sequence to the receiving device.

[0110] In this embodiment, the transmitting end can transmit the electrical signal to be sent from the transmitting end to the receiving end through a wireless transmission device, for example, through WI-FI, Bluetooth, Near Field Communication (NFC), 5G, etc.

[0111] As an optional embodiment, before selecting key frames from the video frame sequence to be transmitted according to the characteristics of the video frame sequence to be transmitted and the channel conditions to obtain the key frame sequence, the method also includes: performing video capture, determining the frame rate and resolution of the video, and obtaining the video frame sequence.

[0112] In this embodiment, the device for video acquisition may be a sensor device for acquiring video in an environment and determining parameters related to the video. One or more cameras may be used to acquire video.

[0113] The transmitting device performs video acquisition, determines the frame rate and resolution of the video, and obtains a video frame sequence. The video frame sequence is subjected to key frame selection to obtain a key frame sequence. The key frame sequence is subjected to semantic extraction to obtain a semantic feature map; the semantic feature map is subjected to source-channel joint coding to obtain a coding sequence; the coding sequence is quantized to obtain a quantized sequence, and the quantized sequence is sent to the receiving device, and the quantized sequence is the quantized coding sequence.

[0114] Among them, the step of the transmitting device performing key frame selection includes: determining an adaptive transmission rate according to the characteristics of the video frame sequence to be transmitted and the channel conditions; determining the number and position of the key frame selection according to the adaptive transmission rate to obtain the key frame sequence.

[0115] After receiving the quantized sequence, the receiving end device decodes it to obtain a reconstructed semantic feature map; performs semantic restoration on the reconstructed semantic feature map to obtain a reconstructed key frame sequence; performs video interpolation on the reconstructed key frame sequence to obtain a complete reconstructed video frame sequence. Optionally, after the video interpolation step, video display is performed to play the reconstructed video frame sequence at a frame rate.

[0116] The step of performing video frame insertion on the reconstructed key frame sequence by the receiving end device includes: determining the number and position of video insertion frames according to the number and position of the selected key frames to obtain a complete reconstructed video frame sequence.

[0117] Specifically, the receiving end can restore the received electrical signal to the reconstructed video through the semantic decoding device, decode the received electrical signal, restore the reconstructed key frame sequence, and then restore the complete video through the reconstructed key frame sequence. The receiving end can also play the reconstructed video through the output device and can play it on the display.

[0118] In this embodiment, the transmitting end determines the channel condition through a spectrum analysis device and collects video through a sensor device. The video is converted into an electrical signal through a semantic encoding device and transmitted through a wireless transmission device. The video is restored and reconstructed through a semantic decoding device at the receiving end and played on a display, thereby realizing wireless video semantic communication, greatly reducing processing delays, improving transmission efficiency, and ensuring communication reliability.

[0119] In order to make the objectives, technical solutions and advantages of the present application more clear, the video communication method of the embodiment of the present application is described below in combination with specific embodiments and with reference to the accompanying drawings.

[0120] The video communication method of the embodiment of the present application is as follows: Figure 2 As shown, for the video X to be transmitted, the transmitter selects key frames from the video frame sequence through a key frame selector to obtain a key frame sequence; the semantic extractor performs semantic extraction on the key frame sequence to obtain a semantic feature map; the semantic feature map is jointly encoded with the source channel to obtain a coding sequence; the coding sequence is quantized by a quantizer, and the quantized coding sequence is sent to the receiver through a wireless channel. It should be noted that before selecting the key frame, the video characteristics such as the bit rate and resolution of the video are obtained.

[0121] The receiving end device decodes the received sequence through a source channel joint decoder to obtain a reconstructed semantic feature map; performs semantic restoration on the reconstructed semantic feature map through a semantic restorer to obtain a reconstructed key frame sequence; and performs video interpolation on the reconstructed key frame sequence through an interpolator to obtain a complete reconstructed video frame sequence.

[0122] Example 1:

[0123] Description of the video to be transmitted.

[0124] In this embodiment, it is assumed that the transmitted video uses the video in the UVG dataset, and each video contains 600 video frames. The UVG dataset is known for its high-quality video materials and wide scene diversity, which provides us with an ideal testing environment. For the frame rate of the test video, it is set to 25fps, which is a common video frame rate standard used in movies, television, and many other video applications. In addition, the resolution is set to 1920×1080, that is, the high-definition 1080p standard, to ensure that the test video has sufficient details and clarity.

[0125] For the transmitting end device, the video communication method comprises the following steps:

[0126] Step 1: Train the transfer model.

[0127] In the model training process of this embodiment, the Vimeo-90k data set is used as the training set, which provides a wide range of video materials and is suitable for training tasks such as video frame reconstruction. The data set contains 89,800 video clips, each of which shows various scenes and actions, and each clip consists of a sequence of 7 frames. During the training process, each frame in each video clip is input as an independent video frame. In order to enhance the generalization ability and robustness of the model, the input frames are randomly cropped during training and cropped to a size of 256×256 pixels. Since image blocks of different sizes and positions need to be processed, this data enhancement technique helps the model learn more robust feature representations. During training, the hyperparameter is set to 8192. In addition, the minimum batch size (mini-batch size) is set to 36. The choice of batch size is crucial to the stability and efficiency of training. A smaller batch size may cause training instability, while a larger batch size may require more computing resources. Setting the minimum batch size to 36 can ensure that the training process is both stable and efficient.

[0128] Step 2: Spectrum analysis.

[0129] This step determines the channel conditions, including determining the channel bandwidth, signal-to-noise ratio, etc.

[0130] Step 3: Determine the adaptive transmission rate.

[0131] The adaptive transmission rate is determined by both the characteristics of the video and the channel conditions. On the one hand, when the channel conditions are good, the channel capacity is large, and the code length can be appropriately increased to increase the transmission rate, but it is always necessary to ensure that the transmission rate is lower than the channel capacity; when the channel conditions are bad, the channel capacity decreases, and the code length should be reduced to reduce the transmission rate to ensure the reliability of communication. On the other hand, the frame rate, resolution, etc. of the video source itself will also affect the transmission rate. In this example, the adaptive transmission rate is controlled by changing the inter-frame compression multiple, that is, by the number of key frame selections.

[0132] According to Shannon's second theorem, the channel capacity of a communication system is Where W is the channel bandwidth, is the signal-to-noise ratio, and its conversion formula is When the transmission rate R>C, reliable transmission cannot be achieved, so it is necessary to ensure that the bit rate R is lower than the channel capacity. In this embodiment, it is assumed that the communication bandwidth is fixed to 1MHz, the frame rate of the video source is assumed to be r=25fps, the size (resolution) is h×w=1920×1080, the inter-frame compression multiple of the key frame selector is assumed to be ρ, and the intra-frame compression multiple of the source channel joint encoder is θ. The transmission rate R can be expressed as: R=r×w×h÷ρ÷θ. The minimum inter-frame compression multiple length ρ can be solved by R≤C min Then, the minimum inter-frame compression multiple length ρ min Calculate the number k of keyframe selections.

[0133] Step 4: Select the key frame to be transmitted.

[0134] In this embodiment, the input of the system is a set of video frame sequences, X = {x1, x2, ..., x T}, a total of T video frames, where the frame at time step t is modeled as a vector of pixel intensities This embodiment selects the key frame sequence of the number k of the key frame selection obtained by determining the adaptive transmission rate by an equal interval sampling method, which is expressed as

[0135] Step 5: Semantic extraction.

[0136] The key frame sequence is semantically extracted through a convolutional network and transformed into a semantic feature map The convolutional network is for example Figure 3 shown.

[0137] Step 6: Source-channel joint coding.

[0138] The semantic feature map is transformed into a continuous value symbol sequence by performing source-channel joint encoding through a convolutional network. The convolutional network is for example Figure 4 shown.

[0139] Step 7: Quantification.

[0140] The continuous value symbol sequence is transformed into a discrete value sequence to be transmitted with a finite value through soft quantization.

[0141] Step 8: Wireless transmission.

[0142] This embodiment mainly considers the widely used additive white Gaussian noise (AWGN) channel. In the AWGN channel, the channel transfer function is: where σ 2 is the energy of the noise. Therefore, the sequence to be transmitted is transmitted through the wireless channel, and the received sequence is obtained

[0143] For the receiving end device, the video communication method comprises the following steps:

[0144] Step 9: The receiving device performs source channel joint decoding.

[0145] The receiving device performs source-channel joint decoding on the received sequence and transforms it into a reconstructed semantic feature map: The convolutional network used for decoding is Figure 5 shown.

[0146] Step 10: The receiving device performs semantic recovery on the received sequence.

[0147] Perform semantic restoration on the reconstructed semantic feature map and transform it into a reconstructed key frame sequence The convolutional network used for semantic restoration is as follows Figure 6 shown.

[0148] Step 11: The receiving device performs video frame insertion on the key frame sequence.

[0149] This embodiment uses an intermediate feature refinement network (IFRNet) as a video frame interpolation network to interpolate virtual frames to approximate the original video frames. Figure 7 As shown, assuming that a key frame is selected every 8 frames, firstly, the key frames are reconstructed by two adjacent and Insert frame Then by and as well as and Insert frames separately and And so on, finally the entire reconstructed video frame sequence is obtained.

[0150] Step 12: After obtaining the reconstructed video frame sequence, the receiving device can play the video at the frame rate of the video.

[0151] The embodiment of the present application can also evaluate the quality of the reconstructed video. For example: This embodiment uses Peak signal-to-noise ratio (PSNR) and Multi Scale Structural Similarity (MS-SSIM) as reconstruction quality evaluation indicators for evaluation. PSNR measures the objective difference between the restored image and the original image, can better reflect the objective quality of the image, and measure the accuracy of frame reconstruction; MS-SSIM takes into account the human visual system's perception of structural information, can better reflect the details and texture information of the image, and measure and evaluate the perceptual quality of the restored frame.

[0152] This embodiment is compared with the traditional video coding transmission scheme and the Model Division Video Semantic Communication (MDVSC) scheme in terms of reconstruction quality and delay. The traditional scheme includes source coding H.264 and H.265, channel coding LDPC with a coding efficiency of 1 / 2, and modulation using binary phase shift keying (BPSK) and 16-bit quadrature amplitude modulation (QAM). Figure 8 , Fig. 9 As shown, WAFI-VSC represents the legend of this example. Under the PSNR index, compared with the 1 / 2LDPC+BPSK scheme, under high signal-to-noise ratio, the index of this embodiment exceeds the scheme combined with H.264, but is slightly lower than the scheme combined with H.265. Under low signal-to-noise ratio, the traditional scheme suffers from the cliff effect, while this embodiment can effectively deal with the cliff effect and obtain better recovery indicators. However, under the MS-SSIM index, this embodiment almost completely surpasses the 1 / 2LDPC+BPSK scheme combining H264 and H265. This shows that this embodiment performs better in indicators that are closer to human perception and has a recovery effect that is more suitable for the human visual system. As for the 1 / 2LDPC+16QAM scheme, affected by the cliff effect, it cannot transmit video normally in the entire SNR range.

[0153] like Figure 8 , Fig. 9 As shown, it can be seen that the PSNR and MS-SSIM curves of this embodiment almost completely surpass MDVSC. In addition, during the test, we added a clock timing function to calculate the processing delay. The test results show that the processing delay per frame of this embodiment does not exceed 1ms, which is only 1 / 1800 of MDVSC.

[0154] In this embodiment, a video communication method for end-to-end adaptive frame selection of video is provided. By adding an adaptive frame selection method and frame insertion technology to a video semantic communication system, the reliability of the video semantic communication system can be ensured while improving the effectiveness of the video semantic communication system and reducing processing delays, thereby achieving better communication effects.

[0155] Example 2: Taking the application of the video communication method of the present application to an unmanned workshop as an example, the effect of video sharing between two unmanned workshops can be achieved, for example: unmanned vehicle A is used as a transmitting device and unmanned vehicle B is used as a receiving device. Compared with Example 1, the following part is added at the beginning of the whole process.

[0156] Video acquisition: Unmanned vehicle A collects real-time video, obtains the video frame sequence to be transmitted, and obtains parameters such as the frame rate and resolution of the video.

[0157] The other contents of this embodiment are similar to those of the above-mentioned Example 1 and will not be elaborated here.

[0158] In the embodiments of the present application, video communication adds a process of removing inter-frame redundancy compared to image communication. This process mostly relies on traditional video communication architecture or other complex processing, and has a high processing delay. AI technology has developed rapidly in recent years, and video interpolation technology has also developed accordingly. Video interpolation is an important underlying computer vision task, and traditional video interpolation mostly relies on optical flow estimation. The present application makes full use of the advantages of video interpolation technology and applies it to video semantic communication. At the sending end, only a small number of selected key frames need to be sent, and the key frames are deeply JSCC encoded and transmitted. At the receiving end, other non-key frames are generated by the restored key frames through the video interpolation network. The present application improves transmission efficiency and saves communication bandwidth through key frame selection and frame generation methods, greatly reduces complexity, and reduces processing delay at the sending end.

[0159] The method for adaptive transmission rate proposed in this embodiment is that the system can intelligently select or adjust the number of selected key frames in real time according to the current channel conditions and the characteristics of the video content itself. This method can avoid data loss or transmission interruption caused by the transmission rate exceeding the actual bandwidth of the channel during data transmission, thereby ensuring the stability and reliability of the communication process. At the same time, by adjusting the number of selected key frames, communication bandwidth resources can be saved to the greatest extent and transmission efficiency can be optimized. This design concept is consistent with the core idea of ​​Shannon's Second Theorem, that is, when the data transmission rate exceeds the bandwidth limit of the channel, it will become impossible to achieve reliable data transmission. Therefore, the method for adaptive transmission rate of the embodiment of the present application not only meets the requirements of communication reliability, but also realizes efficient utilization of communication resources and improves the flexibility of the communication system.

[0160] In addition, 6G technology aims to achieve more stable and reliable communication, and the requirements for latency are also more stringent, especially in scenarios with extremely high requirements for data transmission, such as remote medical surgery and autonomous driving, which need to ensure the accuracy, integrity and immediacy of data transmission. However, traditional methods cannot provide sufficient redundancy and recovery mechanisms to deal with errors and losses in the data transmission process when dealing with high reliability requirements, and cannot provide sufficiently fast transmission speeds and responsiveness. In the case of poor channel conditions, it will also be affected by the cliff effect. With the continuous strengthening of DNN, DNN technology can be used to process the coding process in communication, and source coding and channel coding can be jointly processed. The video to be transmitted is used as the input of the DNN network. Through a series of convolution processing of the DNN network, the size is reduced and the dimension is increased to obtain the final coded sequence to be transmitted. The original video can be restored at the receiving end through the opposite operation. The parameters of the network that are more adapted to the channel can be obtained through training and learning, and a better recovery effect can be obtained. The method of the present application can solve the cliff effect problem in traditional communications and improve the reconstruction quality of the video.

[0161] like Fig.10 As shown, the embodiment of the present application also provides a video communication method, which is applied to a receiving device, including:

[0162] Step 101, receiving a key frame sequence;

[0163] Step 102: performing source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence;

[0164] Step 103: Perform video interpolation on the reconstructed key frame sequence according to the number and positions of the key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence; wherein the number and positions of the key frames are related to the characteristics of the video and the channel conditions.

[0165] In this embodiment, the transmitting end device selects some key frames from the video frame sequence to be transmitted, processes them, and then sends them to the receiving end device, wherein the selected key frames are related to the characteristics of the video and the channel conditions.

[0166] The transmitting device performs video acquisition, determines the frame rate and resolution of the video, and obtains a video frame sequence. The video frame sequence is subjected to key frame selection to obtain a key frame sequence. The key frame sequence is subjected to semantic extraction to obtain a semantic feature map; the semantic feature map is subjected to source-channel joint coding to obtain a coding sequence; the coding sequence is quantized to obtain a quantized sequence, and the quantized sequence is sent to the receiving device. The quantized sequence is the quantized key frame coding sequence.

[0167] After receiving the quantized sequence, the receiving end device decodes it to obtain a reconstructed key frame sequence, and performs video interpolation on the reconstructed key frame sequence according to the number and position of key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence.

[0168] In an embodiment of the present application, a transmitting device selects key frames based on the characteristics of the video itself and the channel conditions, and encodes the key frames and sends them to a receiving device. After receiving the key frame sequence, the receiving device decodes it to obtain a reconstructed key frame sequence, and inserts video frames into the reconstructed key frame sequence according to the number and position of the key frames to obtain a reconstructed video frame sequence. The adaptive key frame selection method can ensure the reliability of communication. During video communication, only the encoded key frames are transmitted, and the receiving end decodes and inserts the key frames to obtain a reconstructed video frame sequence, which reduces the processing delay of the video frames, saves channel bandwidth, and improves transmission efficiency.

[0169] As an optional embodiment, according to the number and positions of key frames in the reconstructed key frame sequence, video interpolation is performed on the reconstructed key frame sequence to obtain a reconstructed video frame sequence, including:

[0170] The position and number of video interpolation frames are determined according to the position and number of key frames in the reconstructed key frame sequence; and video interpolation frames are performed on the reconstructed key frame sequence according to the position and number of video interpolation frames to obtain a reconstructed video frame sequence.

[0171] In this embodiment, the receiving end device determines the number and position of video interpolation frames according to the number and position of key frames to obtain a complete reconstructed video frame sequence. This operation process is simple and easy to implement, reduces the complexity of the video frame processing process, and saves processing resources.

[0172] Optionally, the characteristics of the video include a frame rate and / or a resolution of the video;

[0173] The channel conditions include: channel bandwidth and / or signal-to-noise ratio;

[0174] The number and position of the key frames are determined according to the transmission rate and channel conditions; the transmission rate is calculated according to the frame rate and resolution of the video.

[0175] Specifically, the calculation formula of the transmission rate R is as follows: R = r × w × h ÷ ρ ÷ θ. Among them, r is the frame rate of the video, w × h is the resolution of the video, ρ is the inter-frame compression multiple of the key frame selector, and θ is the intra-frame compression multiple of the source channel joint encoder. The inter-frame compression multiple can be used to indicate how many frames a key frame is selected.

[0176] Since the transmission rate R is less than or equal to the channel capacity C, according to the calculation formula of the above transmission rate R, the minimum inter-frame compression multiple ρ can be solved min , the total number of video frames and the minimum inter-frame compression multiple length ρ are known min , the number of key frame selections k can be calculated.

[0177] Among them, the channel capacity Where W is the channel bandwidth, is the signal-to-noise ratio, and its conversion formula is When the transmission rate R>C, reliable transmission cannot be achieved, so it is necessary to ensure that the code rate R is lower than the channel capacity C.

[0178] As an optional embodiment, performing source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence includes:

[0179] The key frame sequence is subjected to source-channel joint decoding to obtain a reconstructed semantic feature map; and the reconstructed semantic feature map is subjected to semantic restoration to obtain a reconstructed key frame sequence.

[0180] Optionally, the method further includes: playing the reconstructed video frame sequence at a frame rate of the video.

[0181] In this embodiment, the transmitting device performs video acquisition, determines the frame rate and resolution of the video, and obtains a video frame sequence. The video frame sequence is subjected to key frame selection to obtain a key frame sequence. The key frame sequence is subjected to semantic extraction to obtain a semantic feature map; the semantic feature map is subjected to source-channel joint coding to obtain a coding sequence; the coding sequence is quantized to obtain a quantized sequence, and the quantized sequence is sent to the receiving device, and the quantized sequence is the quantized coding sequence.

[0182] After receiving the quantized sequence, the receiving end device decodes it to obtain a reconstructed semantic feature map; performs semantic restoration on the reconstructed semantic feature map to obtain a reconstructed key frame sequence; performs video interpolation on the reconstructed key frame sequence to obtain a complete reconstructed video frame sequence. Optionally, after the video interpolation step, video display is performed to play the reconstructed video frame sequence at a frame rate.

[0183] The step of performing video frame insertion on the reconstructed key frame sequence by the receiving end device includes: determining the number and position of video insertion frames according to the number and position of the selected key frames to obtain a complete reconstructed video frame sequence.

[0184] Specifically, the receiving end can restore the received electrical signal to the reconstructed video through the semantic decoding device, decode the received electrical signal, restore the reconstructed key frame sequence, and then restore the complete video through the reconstructed key frame sequence. The receiving end can also play the reconstructed video through the output device and can play it on the display.

[0185] In this embodiment, the transmitting end determines the channel condition through a spectrum analysis device and collects video through a sensor device. The video is converted into an electrical signal through a semantic encoding device and transmitted through a wireless transmission device. The video is restored and reconstructed through a semantic decoding device at the receiving end and played on a display, thereby realizing wireless video semantic communication, greatly reducing processing delays, improving transmission efficiency, and ensuring communication reliability.

[0186] like Fig.11 As shown, the embodiment of the present application further provides a video communication device 1100, which is applied to a transmitting end device, including:

[0187] The key frame selection module 1110 is used to select key frames from the video frame sequence to be transmitted according to the characteristics of the video and the channel conditions to obtain a key frame sequence;

[0188] The coding module 1120 is used to perform source-channel joint coding on the key frame sequence;

[0189] The wireless transmission module 1130 is used to send the encoded key frame sequence to a receiving device.

[0190] Optionally, the key frame selection module includes:

[0191] A transmission rate determination unit, configured to determine a transmission rate according to characteristics of the video;

[0192] A key frame information determination unit, configured to determine the number and positions of key frames according to the transmission rate and channel conditions;

[0193] The key frame selection unit is used to select key frames from the video frame sequence according to the quantity and position to obtain a key frame sequence.

[0194] Optionally, the characteristics of the video include: frame rate and / or resolution of the video;

[0195] The transmission rate determination unit is specifically used for:

[0196] The transmission rate is calculated based on the video's frame rate, resolution, inter-frame compression multiples of selected key frames, and intra-frame compression multiples of the encoding.

[0197] Optionally, the channel condition includes: channel bandwidth and / or signal-to-noise ratio;

[0198] The key frame information determination unit is specifically used for:

[0199] Determine a minimum inter-frame compression multiple for selecting a key frame according to the transmission rate and the channel capacity, wherein the transmission rate is less than or equal to the channel capacity, and the channel capacity is calculated according to the channel bandwidth and the signal-to-noise ratio;

[0200] Calculating the number of key frames according to the minimum inter-frame compression multiple;

[0201] The position of the key frame in the video frame sequence is determined according to the number of the key frames and a preset sampling method.

[0202] Optionally, the encoding module is specifically used for:

[0203] Performing semantic extraction on the key frame sequence to obtain a semantic feature map of the key frame sequence;

[0204] Perform source-channel joint coding on the semantic feature map to obtain a coding sequence.

[0205] Optionally, the device further comprises:

[0206] A quantization module, used for quantizing the coding sequence;

[0207] The wireless transmission module is specifically used to send the quantized coding sequence to a receiving device.

[0208] Optionally, the device further comprises:

[0209] The video acquisition module is used to acquire video, determine the frame rate and resolution of the video, and obtain the video frame sequence.

[0210] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiment applied to the sending end device, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0211] like Fig.12 As shown, the embodiment of the present application further provides a video communication device 1200, which is applied to a receiving end device, including:

[0212] The receiving module 1210 is used to receive a key frame sequence;

[0213] A decoding module 1220, configured to perform source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence;

[0214] A video frame insertion module 1230 is used to perform video frame insertion on the reconstructed key frame sequence according to the number and positions of the key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence;

[0215] The number and position of the key frames are related to the characteristics of the video and the channel conditions.

[0216] Optionally, the video frame insertion module is specifically used for:

[0217] Determining the position and number of video interpolation frames according to the position and number of key frames in the reconstructed key frame sequence;

[0218] Video frame insertion is performed on the reconstructed key frame sequence according to the position and number of the video frame insertion to obtain a reconstructed video frame sequence.

[0219] Optionally, the characteristics of the video include a frame rate and / or a resolution of the video;

[0220] The channel conditions include: channel bandwidth and / or signal-to-noise ratio;

[0221] The number and position of the key frames are determined according to the transmission rate and channel conditions; the transmission rate is calculated according to the frame rate and resolution of the video.

[0222] Optionally, the decoding module is specifically used for:

[0223] Performing source-channel joint decoding on the key frame sequence to obtain a reconstructed semantic feature map;

[0224] Perform semantic restoration on the reconstructed semantic feature map to obtain a reconstructed key frame sequence.

[0225] Optionally, the device further comprises:

[0226] The display module is used to play the reconstructed video frame sequence according to the frame rate of the video.

[0227] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented by the above-mentioned method embodiment applied to the receiving end device, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0228] like Fig.13 As shown, an electronic device of an embodiment of the present application, which can be a sending end device or a receiving end device, includes a transceiver 1310, a processor 1300, a memory 1320, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the steps of the above-mentioned video communication method are implemented, and the transceiver 1310 is used to receive and send data under the control of the processor 1300.

[0229] Wherein, when the electronic device is a transmitting end device, the processor 1300 is configured to:

[0230] According to the characteristics of the video and the channel conditions, key frames are selected from the video frame sequence to be transmitted to obtain a key frame sequence;

[0231] Performing source-channel joint coding on the key frame sequence;

[0232] The transceiver 1310 is used to send the encoded key frame sequence to a receiving device.

[0233] Optionally, the processor 1300 selects key frames from a video frame sequence to be transmitted according to video characteristics and channel conditions to obtain a key frame sequence, including:

[0234] Determine the transmission rate based on the characteristics of the video;

[0235] Determining the number and positions of key frames according to the transmission rate and channel conditions;

[0236] Key frames are selected from the video frame sequence according to the quantity and position to obtain a key frame sequence.

[0237] Optionally, the characteristics of the video include: frame rate and / or resolution of the video;

[0238] The processor 1300 determines the transmission rate according to the characteristics of the video, including:

[0239] The transmission rate is calculated based on the video's frame rate, resolution, inter-frame compression multiples of selected key frames, and intra-frame compression multiples of the encoding.

[0240] Optionally, the channel condition includes: channel bandwidth and / or signal-to-noise ratio;

[0241] The processor 1300 determines the number and position of key frames according to the transmission rate and channel conditions, including:

[0242] Determine a minimum inter-frame compression multiple for selecting a key frame according to the transmission rate and the channel capacity, wherein the transmission rate is less than or equal to the channel capacity, and the channel capacity is calculated according to the channel bandwidth and the signal-to-noise ratio;

[0243] Calculating the number of key frames according to the minimum inter-frame compression multiple;

[0244] The position of the key frame in the video frame sequence is determined according to the number of the key frames and a preset sampling method.

[0245] Optionally, the processor 1300 performs source-channel joint coding on the key frame sequence, including:

[0246] Performing semantic extraction on the key frame sequence to obtain a semantic feature map of the key frame sequence;

[0247] Perform source-channel joint coding on the semantic feature map to obtain a coding sequence.

[0248] Optionally, the processor 1300 is further configured to:

[0249] quantifying the coding sequence;

[0250] The transceiver is specifically used for:

[0251] Send the quantized code sequence to the receiving device.

[0252] Optionally, the processor 1300 is further used to: perform video capture, determine the frame rate and resolution of the video, and obtain a video frame sequence.

[0253] In the case where the electronic device is a receiving end device, the transceiver 1310 is used to: receive a key frame sequence;

[0254] The processor 1300 is used to: perform source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence;

[0255] Performing video frame insertion on the reconstructed key frame sequence according to the number and positions of the key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence;

[0256] The number and position of the key frames are related to the characteristics of the video and the channel conditions.

[0257] Optionally, the processor 1300 performs video frame insertion on the reconstructed key frame sequence according to the number and positions of key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence, including:

[0258] Determining the position and number of video interpolation frames according to the position and number of key frames in the reconstructed key frame sequence;

[0259] Video frame insertion is performed on the reconstructed key frame sequence according to the position and number of the video frame insertion to obtain a reconstructed video frame sequence.

[0260] Optionally, the characteristics of the video include a frame rate and / or a resolution of the video;

[0261] The channel conditions include: channel bandwidth and / or signal-to-noise ratio;

[0262] The number and position of the key frames are determined according to the transmission rate and channel conditions; the transmission rate is calculated according to the frame rate and resolution of the video.

[0263] Optionally, the processor 1300 performs source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence, including:

[0264] Performing source-channel joint decoding on the key frame sequence to obtain a reconstructed semantic feature map;

[0265] Perform semantic restoration on the reconstructed semantic feature map to obtain a reconstructed key frame sequence.

[0266] Optionally, the processor 1300 is further configured to:

[0267] The reconstructed video frame sequence is played at the frame rate of the video.

[0268] Among them, Fig.13 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1300 and memory represented by memory 1320. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1310 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 1300 is responsible for managing the bus architecture and general processing, and the memory 1320 may store data used by the processor 1300 when performing operations.

[0269] A readable storage medium of an embodiment of the present application stores a program or instruction thereon. When the program or instruction is executed by a processor, the steps in the video communication method described above are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0270] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0271] An embodiment of the present application also provides a computer program product on which programs or instructions are stored, including computer instructions. When the computer instructions are executed by a processor, the steps in the video communication method described above are implemented and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0272] It should be further explained that the terminal devices described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

[0273] In the embodiment of the present application, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0274] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0275] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0276] The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present application. Therefore, the present application should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present application will be perfect and complete, and the scope of the present application will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the text clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to include these multiple forms. It will be further understood that the terms "comprising" and / or "including" when used in this specification indicate the presence of the features, integers, steps, operations, components and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and / or their groups. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of that range and any subranges therebetween.

[0277] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video communication method, characterized in that: Applied to the sending device, including: According to the characteristics of the video and the channel conditions, key frames are selected from the video frame sequence to be transmitted to obtain a key frame sequence; Performing source-channel joint coding on the key frame sequence; The encoded key frame sequence is sent to a receiving device.

2. The method according to claim 1, characterized in that According to the characteristics of the video and the channel conditions, key frames are selected from the video frame sequence to be transmitted to obtain a key frame sequence, including: Determine the transmission rate based on the characteristics of the video; Determining the number and positions of key frames according to the transmission rate and channel conditions; Key frames are selected from the video frame sequence according to the quantity and position to obtain a key frame sequence.

3. The method according to claim 2, characterized in that The characteristics of the video include: frame rate and / or resolution of the video; Determining the transmission rate according to the characteristics of the video includes: The transmission rate is calculated based on the video's frame rate, resolution, inter-frame compression multiples of selected key frames, and intra-frame compression multiples of the encoding.

4. The method according to claim 2, characterized in that: The channel conditions include: channel bandwidth and / or signal-to-noise ratio; Determining the number and position of key frames according to the transmission rate and channel conditions includes: Determine a minimum inter-frame compression multiple for selecting a key frame according to the transmission rate and the channel capacity, wherein the transmission rate is less than or equal to the channel capacity, and the channel capacity is calculated according to the channel bandwidth and the signal-to-noise ratio; Calculating the number of key frames according to the minimum inter-frame compression multiple; The position of the key frame in the video frame sequence is determined according to the number of the key frames and a preset sampling method.

5. The method according to claim 1, characterized in that Performing source-channel joint coding on the key frame sequence includes: Performing semantic extraction on the key frame sequence to obtain a semantic feature map of the key frame sequence; Perform source-channel joint coding on the semantic feature map to obtain a coding sequence.

6. The method according to claim 5, characterized in that Before sending the encoded key frame sequence to the receiving end device, the method further includes: quantifying the coding sequence; The step of sending the encoded key frame sequence to a receiving device comprises: Send the quantized code sequence to the receiving device.

7. The method according to claim 1, characterized in that Before selecting key frames from the video frame sequence to be transmitted according to the characteristics of the video frame sequence to be transmitted and the channel conditions to obtain the key frame sequence, the method further includes: Perform video capture, determine the frame rate and resolution of the video, and obtain the video frame sequence.

8. A video communication method, characterized in that: Applied to receiving devices, including: receiving a key frame sequence; Performing source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence; Performing video frame insertion on the reconstructed key frame sequence according to the number and positions of the key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence; The number and position of the key frames are related to the characteristics of the video and the channel conditions.

9. The method according to claim 8, characterized in that According to the number and positions of the key frames in the reconstructed key frame sequence, video interpolation is performed on the reconstructed key frame sequence to obtain a reconstructed video frame sequence, including: Determining the position and number of video interpolation frames according to the position and number of key frames in the reconstructed key frame sequence; Video frame insertion is performed on the reconstructed key frame sequence according to the position and number of the video frame insertion to obtain a reconstructed video frame sequence.

10. The method according to claim 8, characterized in that The characteristics of the video include the frame rate and / or resolution of the video; The channel conditions include: channel bandwidth and / or signal-to-noise ratio; The number and position of the key frames are determined according to the transmission rate and channel conditions; the transmission rate is calculated according to the frame rate and resolution of the video.

11. The method according to claim 8, characterized in that Performing source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence, including: Performing source-channel joint decoding on the key frame sequence to obtain a reconstructed semantic feature map; Perform semantic restoration on the reconstructed semantic feature map to obtain a reconstructed key frame sequence.

12. The method according to claim 8, characterized in that The method further comprises: The reconstructed video frame sequence is played at the frame rate of the video.

13. A video communication device, characterized in that: include: A key frame selection module is used to select key frames from the video frame sequence to be transmitted according to the characteristics of the video and the channel conditions to obtain a key frame sequence; A coding module, used for performing source-channel joint coding on the key frame sequence; The wireless transmission module is used to send the encoded key frame sequence to a receiving device.

14. A video communication device, characterized in that: include: A receiving module, used for receiving a key frame sequence; A decoding module, used for performing source-channel joint decoding on the key frame sequence to obtain a reconstructed key frame sequence; A video frame insertion module, configured to perform video frame insertion on the reconstructed key frame sequence according to the number and positions of the key frames in the reconstructed key frame sequence to obtain a reconstructed video frame sequence; The number and position of the key frames are related to the characteristics of the video and the channel conditions.

15. An electronic device, characterized in that: It comprises a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, it implements the steps of the video communication method as described in any one of claims 1 to 7, or implements the steps of the video communication method as described in any one of claims 8 to 12.

16. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the video communication method according to any one of claims 1 to 7 are implemented, or the steps of the video communication method according to any one of claims 8 to 12 are implemented.

17. A computer program product having a program or instruction stored thereon, characterized in that: The method comprises computer instructions, which, when executed by a processor, implement the steps of the video communication method according to any one of claims 1 to 7, or implement the steps of the video communication method according to any one of claims 8 to 12.

Images