Video transmission method and device, equipment, storage medium and product
By calculating the color histogram difference and SSIM of video frames, important frames are identified and transmitted, solving the problem of poor video continuity in weak network environments and achieving high-quality video transmission.
Patent Information
- Application Number
- CN202511676933.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-02-13
AI Technical Summary
In low bitrate application scenarios, existing audio and video transmission methods cannot effectively improve video continuity in weak network environments, resulting in video quality jitter and network congestion.
By calculating the color histogram difference and structural similarity index (SSIM) between the target frame and the previous frame, unimportant frames are identified and skipped, and only frames carrying important information are transmitted, in order to adapt to channel bandwidth and improve video continuity.
It improves the continuity and quality of video transmission, reduces bandwidth usage and video stuttering, is highly adaptable, and has low transformation costs.
Smart Images
Figure CN121531129A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, and in particular to a video transmission method and device, equipment, storage medium and computer program product. BACKGROUND
[0002] In the current audio and video encoding and transmission process, the collected audio and video are encoded by the traditional rate control (RC) mechanism of the H.264 encoding standard, and then transmitted to the terminal, i.e. the receiving end, for playing. In the traditional RC mechanism of the H.264 encoding standard, the encoding process is based on a fixed constant frame rate, and the core control logic is to adjust the quantization parameter (QP) of the frame layer or the macro block (MB) layer to match the encoding output rate with the channel bandwidth. In the high rate application scenario, the mechanism can better balance the rate and the encoding quality, and realize stable video output, because the bandwidth resource is sufficient. However, in the low rate application scenario, in order to forcibly match the limited bandwidth, the encoding process needs to set a very low spatial quality parameter for the encoded frame, which leads to serious loss of video picture details. At the same time, in order to avoid overflow of the encoding buffer, the mechanism will trigger a passive frame skipping encoding operation. When there is a dramatic motion scene (such as a fast-moving target) or a scene change in the video sequence, the passive frame skipping encoding phenomenon will be significantly aggravated, and then the video received by the receiving end will be discontinuous, and obvious video quality jitter will occur, which seriously affects the user's viewing experience.
[0003] In the existing audio and video anti-weak network transmission method, the strategy of forward error correction coding (FEC) or retransmission of I frames, i.e. key frames, is often used to improve the video continuity. FEC increases redundant data in the transmission data, so that the receiving end can recover the original data through the redundant data when some data packets are lost, reducing the influence of packet loss on video continuity. The retransmission of I frames ensures that the receiving end can normally recover the video sequence by retransmitting the lost key frames, thereby improving the video continuity.
[0004] However, in the Cat.1 communication scenario, limited by the lower uplink peak transmission rate, when the network is in a weak network state (such as the actual uplink bandwidth is much lower than 5 Mbps), the redundant data added by FEC will further occupy the limited bandwidth, resulting in a decrease in the effective data transmission rate; and the retransmission of I frames will occupy a large amount of uplink resources due to the large amount of I frame data, which not only cannot recover the lost video frames in time, but also will exacerbate network congestion, leading to continuous deterioration of the network state, and ultimately cannot effectively improve the continuity of the video transmitted to the receiving end in the weak network environment. SUMMARY
[0005] The embodiment of the present application provides a video transmission method, device, equipment, storage medium and computer program product, which can improve video continuity.
[0006] In a first aspect, the embodiment of the present application provides a video transmission method, comprising: obtaining a first color histogram distribution of a target frame and a second color histogram distribution of a previous frame of the target frame, the target frame being any frame in a video; determining a difference degree of the first color histogram distribution and the second color histogram distribution as a distribution difference degree of the target frame; in a case where the distribution difference degree of the target frame is less than a first threshold, determining a recovery frame of the target frame according to a previous frame of the target frame and a motion vector of the target frame; determining a structural similarity index (SSIM) of the target frame and the recovery frame; in a case where the SSIM is less than a second threshold, determining the target frame as an encoded frame; sending the encoded frame to a receiving end.
[0007] In one possible implementation, in a case where the distribution difference degree of the target frame is less than the first threshold, before determining the recovery frame of the target frame according to the previous frame of the target frame and the motion vector of the target frame, the method further comprises: obtaining distribution difference degrees of n frames adjacent to the target frame and located in front of the target frame, n being an integer not less than 2; determining an average of the distribution difference degrees of the n frames as the first threshold.
[0008] In one possible implementation, in a case where the SSIM is less than the second threshold, before determining the target frame as the encoded frame, the method further comprises: obtaining SSIMs corresponding to k encoded frames adjacent to the target frame and located in front of the target frame, k being an integer not less than 2; determining an average of the SSIMs corresponding to the k encoded frames as the second threshold.
[0009] In one possible implementation, the determination of the SSIM of the target frame and the recovery frame comprises: dividing the target frame and the recovery frame into m regions respectively; selecting a first central region from the m regions corresponding to the target frame and a second central region from the m regions corresponding to the recovery frame; determining the SSIM of the first central region and the second central region.
[0010] In one possible implementation, the method further comprises: in a case where the distribution difference degree of the target frame is not less than the first threshold, determining the target frame as the encoded frame.
[0011] In one possible implementation, when the distribution difference of the target frame is less than a first threshold, the recovered frame of the target frame is determined based on the previous frame of the target frame and the motion vector of the target frame, including: If the distribution difference of the target frame is less than the first threshold, the scaling factor for scaling the multiple encoded frames that are adjacent to and before the target frame to the frame before the target frame is determined based on the inter-frame distance between the multiple encoded frames adjacent to and before the target frame and the target frame. Determine the motion vector of the previous frame of the target frame based on the scaling factor; The motion vector of the previous frame before the target frame is determined as the motion vector of the target frame; The recovered frame of the target frame is determined based on the previous frame of the target frame and the motion vector of the target frame.
[0012] Secondly, embodiments of this application provide a video transmission apparatus, comprising: The acquisition module is used to acquire the first color histogram distribution of the target frame and the second color histogram distribution of the frame preceding the target frame. The target frame is any frame in the video. The determination module is used to determine the difference between the distribution of the first color histogram and the distribution of the second color histogram as the distribution difference of the target frame; The determination module is also used to determine the recovery frame of the target frame based on the previous frame of the target frame and the motion vector of the target frame when the distribution difference of the target frame is less than a first threshold. The determination module is also used to determine the structural similarity index (SSIM) between the target frame and the recovered frame; The determination module is also used to determine that the target frame is a coded frame when the SSIM is less than the second threshold; The sending module is used to send encoded frames to the receiving end.
[0013] Thirdly, embodiments of this application provide an electronic device, the device comprising: Processor and memory storing computer program instructions; A video transmission method that implements any of the above-mentioned features when the processor executes computer program instructions.
[0014] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement a video transmission method according to any one of the above-mentioned methods.
[0015] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform any of the video transmission methods described above.
[0016] The video transmission method, apparatus, device, storage medium, and computer program product of this application embodiment acquire a first color histogram distribution of a target frame and a second color histogram distribution of the frame preceding the target frame, wherein the target frame is any frame in the video; determine the difference between the first color histogram distribution and the second color histogram distribution as the distribution difference of the target frame; if the distribution difference of the target frame is less than a first threshold, determine a restored frame of the target frame based on the previous frame of the target frame and the motion vector of the target frame; determine the structural similarity index (SSIM) between the target frame and the restored frame; if the SSIM is less than a second threshold, determine that the target frame is an encoded frame; and send the encoded frame to the receiving end.
[0017] By calculating the color histogram difference between the target frame and the previous frame, the similarity of the two frames is determined. If the difference is less than a first threshold, indicating that the target frame is similar to the previous frame, the structural similarity index between the target frame and its recovered frame is further determined to verify the recovery effect. If the structural similarity index between the target frame and its recovered frame is less than a second threshold, indicating that the recovered frame quality is insufficient and the target frame carries important information, the target frame is marked as an encoded frame and transmitted to ensure that the receiving end can obtain frames with important information, thereby recovering skipped frames with high quality and improving video continuity. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic diagram of a video transmission architecture provided in one embodiment of this application; Figure 2 This is a flowchart illustrating a video transmission method provided in another embodiment of this application; Figure 3 This is a flowchart illustrating a video transmission method provided in yet another embodiment of this application; Figure 4 This is a flowchart illustrating a video transmission method provided in another embodiment of this application; Figure 5 This is a flowchart illustrating a video transmission method provided in another embodiment of this application; Figure 6 This is a schematic diagram of the selection of the central region provided in another embodiment of this application; Figure 7 This is a schematic diagram of obtaining motion vectors provided in another embodiment of this application; Figure 8This is a schematic diagram of the structure of a video transmission device provided in another embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0022] In current audio and video encoding and transmission processes, the acquired audio and video are encoded using the traditional Rate Control (RC) mechanism of the H.264 encoding standard and then transmitted to the terminal (receiving end) for playback via the network. In the traditional RC mechanism of the H.264 encoding standard, the encoding process is based on a fixed constant frame rate. Its core control logic is to adjust the quantization parameter (QP) at the frame layer or macroblock (MB) layer to match the encoded output bitrate with the channel bandwidth. In high-bitrate application scenarios, this mechanism can effectively balance bitrate and encoding quality, achieving stable video output due to ample bandwidth resources. However, in low-bitrate application scenarios, to force matching with limited bandwidth, the encoding process needs to set extremely low spatial quality parameters for the encoded frames, resulting in severe loss of video detail. Simultaneously, to avoid encoding buffer overflow, this mechanism triggers passive frame skipping encoding operations. When there are scenes with violent motion (such as fast-moving targets) or scene changes in the video sequence, the passive frame skipping coding phenomenon will be significantly aggravated, which will cause the video received by the receiver to be discontinuous, resulting in obvious video quality jitter and seriously affecting the user's viewing experience.
[0023] Existing methods for audio and video transmission in weak network conditions often employ Forward Error Correction (FEC) or retransmission of I-frames (keyframes) to improve video continuity. FEC adds redundant data to the transmitted data, allowing the receiver to recover the original data from the lost packets, thus reducing the impact of packet loss on video continuity. Retransmission of I-frames, on the other hand, ensures that the receiver can correctly reconstruct the video sequence by retransmitting the lost keyframes, further improving video continuity.
[0024] However, in Cat.1 communication scenarios, the peak uplink / downlink rates of Cat.1 terminals are 5 / 10 Mbps. Cat.1 communication capabilities are in the low-to-medium speed range, which can meet certain requirements for high-speed mobility, latency sensitivity, low cost, and low power consumption scenarios. Meanwhile, in wireless environments, network conditions can become unstable due to various factors (such as signal strength and network congestion), posing a challenge to audio and video transmission. Limited by the low peak uplink transmission rate, when the network is in a weak state (e.g., the actual uplink bandwidth is far below 5 Mbps), the redundant data added by FEC will further consume the limited bandwidth, leading to a decrease in the effective data transmission rate. Furthermore, retransmitting I-frames, due to their large data size, consumes a significant amount of uplink resources, not only failing to recover lost video frames in a timely manner but also exacerbating network congestion, causing the network condition to deteriorate continuously, ultimately failing to effectively improve the continuity of video transmitted to the receiving end in weak network environments.
[0025] It should be noted that CatX stands for Long Term Evolution User Equipment Category X (LTE UE Category X), a standard used to measure the wireless performance of user terminal equipment. UE Category encompasses many wireless characteristics, the most important of which is the data rate supported by the UE. Cat.1 defines uplink / downlink peak rates of 5 / 10 Mbps.
[0026] To address the problems of existing technologies, embodiments of this application provide a video transmission method, apparatus, device, storage medium, and computer program product. Embodiments of this application determine the similarity of two frames by calculating the color histogram difference between a target frame and the previous frame. If the difference is less than a first threshold, indicating similarity between the target frame and the previous frame, the structural similarity index between the target frame and its recovered frame is further determined to verify the recovery effect. If the structural similarity index between the target frame and its recovered frame is less than a second threshold, indicating insufficient recovered frame quality and that the target frame carries important information, the target frame is marked as an encoded frame and transmitted to ensure that the receiving end can obtain frames containing important information, thereby recovering skipped frames with high quality and improving video continuity.
[0027] The video transmission method provided in this application embodiment can be applied to an adaptive frame skipping system. This adaptive frame skipping system is located in, for example... Figure 1 The video transmission architecture shown is a video acquisition system 110, an adaptive frame skipping system 120, a video encoding system 130, an encoding buffer 140, and a network 150. The acquisition system 110 is used to acquire video. The adaptive frame skipping system 120 is used to filter out encoded frames from the acquired video and send the encoded frames to the video encoding system 130. After encoding the encoded frames, the video encoding system 130 transmits them to the encoding buffer 140 and then transmits them to the receiving end via the network 150. The acquisition system 110 can be located at a monitoring terminal. This embodiment of the application uses an adaptive frame skipping system to actively skip unimportant frames before encoding, while retaining frames containing important information. By using adaptive frame skipping to regulate the output video frame rate to a suitable level, the output video bitstream is made suitable for the channel transmission bandwidth, thus optimizing the video quality at the receiving end and minimizing distortion.
[0028] Furthermore, existing technologies use specific encoders or additional chips to perform fixed-frame-rate frame skipping encoding and transmission of video, which requires significant modifications to existing systems or equipment. The adaptive frame skipping system provided in this application has minimal invasiveness to existing systems and low integration costs into existing equipment terminals. It can also reuse the hardware encoding capabilities of monitoring terminals.
[0029] The video transmission method provided in the embodiments of this application will be described in detail below.
[0030] like Figure 2 As shown, the video transmission method provided in this application embodiment includes the following steps S210 to S260.
[0031] S210. Obtain the first color histogram distribution of the target frame and the second color histogram distribution of the frame preceding the target frame. The target frame is any frame in the video.
[0032] The target frame is any frame in the video to be transmitted.
[0033] In some embodiments, the horizontal axis of the color histogram represents the brightness of a frame in the video, and from left to right, it represents the gradual transition from pure black to pure white. The vertical axis of the color histogram represents the number of pixels within a frame that fall within the brightness range corresponding to the horizontal axis. The target frame and the frame preceding the target frame in the video are obtained. Based on the color histogram, the first color histogram distribution of the target frame and the second color histogram distribution of the frame preceding the target frame are calculated.
[0034] In some embodiments, the color histogram distributions of consecutive temporal frames in the video to be transmitted are sequentially calculated according to the color histograms and stored in a storage unit. Obtaining the first color histogram distribution of the target frame and the second color histogram distribution of the frame preceding the target frame from the storage unit enables rapid acquisition of these distributions.
[0035] S220. Determine the difference between the distribution of the first color histogram and the distribution of the second color histogram as the distribution difference of the target frame.
[0036] In some embodiments, the analysis and judgment are based on the difference in color histogram distributions of corresponding orders between consecutive frames. The difference between the first color histogram distribution and the second color histogram distribution is determined as the distribution difference of the target frame. The first color histogram distribution and the second color histogram distribution have the same order.
[0037] In one example, the difference between the first color histogram distribution and the second color histogram distribution is determined using the following formula (1): (1) Where m is the order of the statistics of the first-color histogram distribution and the second-color histogram distribution, i.e., the number of intervals. The distribution difference of the target frame. This represents the number of pixels in the i-th interval of the target frame. This represents the number of pixels in the i-th interval of the previous frame.
[0038] S230. When the distribution difference of the target frame is less than the first threshold, determine the restored frame of the target frame based on the previous frame of the target frame and the motion vector of the target frame.
[0039] In some embodiments, if the distribution difference of the target frame is less than a first threshold, the target frame and the frame preceding the target frame are considered similar frames. Based on the motion vectors of the previous frame and the target frame, a restored frame for the target frame is determined. The restored frame is then used to re-determine whether the previous frame and the target frame are similar, and whether the target frame can be skipped. The first threshold can be pre-set according to actual needs.
[0040] In some embodiments, color histograms describe the color distribution within a frame, exhibiting considerable tolerance to motion and adaptability to image rotation and viewing angle. However, color histograms lose positional information about colors. Simple color histogram matching can easily lead to false detections of scene transitions, i.e., misclassification of similar frames. Color histogram comparisons are ineffective in the following two situations: First, because color histograms do not consider pixel positional information, two frames with completely different pixel positions may have similar color histogram distributions if they share the same background and the same target. In this case, using color histogram feature values to check for scene transitions can easily result in missed detections, identifying frames with significant differences as similar frames. Second, frames belonging to the same scene may also have different color histograms. For example, a sudden change in brightness, such as turning on a light or using a flash, can misidentify the same scene as different scenes, identifying similar frames as significantly different frames.
[0041] Considering that color histograms can misjudge certain scenarios, a subjective quality attribute is added between the original frame and the restored frame obtained through motion vector copying to determine whether a scene switch has occurred in the video frame. The restored frame and the original frame are subjected to a subjective quality evaluation based on the Structural Similarity Index (SSIM) to adaptively determine whether to skip the target frame.
[0042] In some embodiments, it is first assumed that the target frame fn is skipped, and the MV field of fn (e.g., in 4×4 blocks) is estimated using the processed motion vector (MV) field of the previous frame. Then, the recovery frame of the target frame is determined based on the previous frame of the target frame and the MV field of the target frame.
[0043] S240. Determine the structural similarity index (SSIM) between the target frame and the recovered frame.
[0044] In some embodiments, a structural similarity index (SSIM) is determined between the target frame and the recovered frame. A larger SSIM indicates that the target frame fn is more likely to be skipped. SSIM is an evaluation metric used to measure the similarity between two digital images; it simulates human visual perception of image quality by comparing the brightness, contrast, and structure of the images.
[0045] S250. If SSIM is less than the second threshold, the target frame is determined to be a coded frame.
[0046] In some embodiments, a larger SSIM indicates that the target frame fn is more likely to be skipped; conversely, if SSIM is less than a second threshold, the target frame is determined to be an encoded frame. The encoded frame is then sent to the receiving end after encoding.
[0047] The second threshold can be pre-set according to actual needs.
[0048] S260, Send an encoded frame to the receiving end.
[0049] In some embodiments, sending an encoded frame to a receiving end includes: sending the encoded frame to a video encoding system, where the video encoding system encodes the encoded frame to obtain a transmission frame; and sending the transmission frame to the receiving end via a network. Upon receiving the transmission frame, the receiving end decodes it to obtain the encoded frame. The receiving end can then reconstruct the video frame based on the encoded frame to achieve continuous video playback.
[0050] This application's embodiments determine whether the content of the two frames is similar by calculating the color histogram difference between the target frame and the previous frame. If the difference is less than a first threshold, meaning the target frame is similar to the previous frame, the structural similarity index between the target frame and its restored frame is further determined to verify the restoration effect. If the structural similarity index between the target frame and its restored frame is less than a second threshold, meaning the restored frame quality is insufficient, indicating that the target frame carries important information, the target frame is marked as an encoded frame and transmitted to ensure that the receiving end can obtain frames with important information, thereby restoring skipped frames with high quality and improving video continuity.
[0051] In some embodiments, such as Figure 3 As shown, before S230 above, the method may also include S221 to S222.
[0052] S221. Obtain the distribution difference of n frames that are adjacent to and before the target frame, where n is an integer not less than 2.
[0053] In some embodiments, a distribution difference sliding window is constructed in the video, and the length of the distribution difference sliding window is n, i.e., n frames. When determining whether the target frame and the frame preceding the target frame are similar, the distribution difference of the n frames in the nearest sliding window preceding the target frame is obtained, thus obtaining the distribution difference of the n frames adjacent to and preceding the target frame, where n is an integer not less than 2.
[0054] In some embodiments, prior to S221 described above, user input n is received.
[0055] Understandably, the size of n affects the robustness and response speed of identifying similar frames.
[0056] In one example, the length n of the distribution difference sliding window is 3, which can achieve a good balance between robustness and response speed.
[0057] S222. The average value of the distribution difference of n frames is determined to be the first threshold.
[0058] In some embodiments, the average distribution difference of n frames within the sliding window period is calculated using a distribution difference sliding window, and this average value is used as a first threshold to determine whether the target frame and the frame preceding the target frame are similar.
[0059] In some embodiments, the distribution variability of the target frame is inserted into a distribution variability sliding window.
[0060] The embodiments of this application use a dynamically changing first threshold to make the judgment of frame similarity more consistent with the scene in which the video is located, thereby improving the accuracy of similar frame recognition.
[0061] In some embodiments, such as Figure 4 As shown, before S250 above, the method may also include S241 to S242.
[0062] S241. Obtain the SSIM corresponding to the k coded frames adjacent to and preceding the target frame, where k is an integer not less than 2.
[0063] In some embodiments, a sliding window is constructed for the k frames that have been encoded before the target frame, and the SSIMs corresponding to the k encoded frames within the sliding window are obtained. This yields the SSIMs corresponding to the k encoded frames that are adjacent to and before the target frame, where k is an integer not less than 2.
[0064] In some embodiments, prior to S241 described above, user input of k is received.
[0065] Understandably, the size k of the sliding window reflects the speed at which the frame skipping mechanism responds to local temporal subjective quality changes in the video sequence. A larger k value means a slower response, while a smaller k value means a faster response.
[0066] In one example, the value of k is set to 2 to achieve a better balance between robustness and reaction speed.
[0067] S242. Determine the average value of the SSIM corresponding to the k encoded frames as the second threshold.
[0068] In some embodiments, the average value of SSIM for each frame in the sliding window is calculated, and this average value is used as the frame skipping reference threshold for the target frame, i.e., the second threshold.
[0069] The embodiments of this application use a dynamically changing second threshold to make the judgment of frame similarity more consistent with the scene in which the video is located, thereby improving the accuracy of similar frame recognition.
[0070] In some embodiments, such as Figure 5 As shown, the above S240 may specifically include S2401 to S2403.
[0071] S2401. Divide the target frame and the recovery frame into m regions respectively.
[0072] In some embodiments, when calculating the SSIM of the target frame and the recovered frame, the SSIM of the entire frame is not calculated. The target frame is divided into m regions, and the recovered frame is divided into m regions. A portion of each region is selected, and the SSIM of that portion is calculated.
[0073] In one example, the target frame and the recovered frame are divided as follows: Figure 6 The area shown is a 3x3 grid of 9 cells.
[0074] S2402. Select the first central region from the m regions corresponding to the target frame, and select the second central region from the m regions corresponding to the recovered frame.
[0075] In some embodiments, since the region of interest of the human eye is concentrated in the center of the video frame, when calculating the SSIM of the restored frame and the target frame, a first central region is selected from the m regions corresponding to the target frame, and a second central region is selected from the m regions corresponding to the restored frame. Only the SSIM of the first central region and the second central region is calculated.
[0076] In one example, select as follows Figure 6 The SSIM of the central cross region shown is calculated for the target frame and the recovered frame to reduce computational complexity.
[0077] S2403, Determine the SSIM of the first central region and the second central region.
[0078] In one example, such as Figure 6 As shown, the SSIM of each region in the first central region and the corresponding region in the second central region are calculated respectively, resulting in multiple sub-SSIMs. The average value of each sub-SSIM is calculated to obtain the SSIM of the first central region and the second central region.
[0079] Based on the characteristic that the region of interest of the human eye is concentrated in the center of the video frame, this embodiment selects only the central region from the target frame and the restored frame, and uses the SSIM of the central region of the target frame and the restored frame as the SSIM of the target frame and the restored frame, thereby reducing the computational complexity and improving the computational efficiency.
[0080] In some embodiments, corresponding to S230 above, the method may further include: If the distribution difference of the target frame is not less than the first threshold, the target frame is determined to be an encoded frame.
[0081] In some embodiments, if the distribution difference of the target frame is not less than a first threshold, it can be considered that the image scene of the target frame may have changed. If a scene change occurs, the target frame should be transmitted, so the target frame is determined to be an encoded frame.
[0082] In this embodiment of the application, when the distribution difference of the target frame is not less than the first threshold, that is, when the difference frame can be identified by the color histogram, there is no need to perform SSIM judgment. The target frame is directly determined to be an encoded frame, reducing the judgment steps and improving efficiency.
[0083] In some embodiments, the above-mentioned S230 may specifically include: If the distribution difference of the target frame is less than the first threshold, the scaling factor for scaling the multiple encoded frames that are adjacent to and before the target frame to the frame before the target frame is determined based on the inter-frame distance between the multiple encoded frames adjacent to and before the target frame and the target frame. Determine the motion vector of the previous frame of the target frame based on the scaling factor; The motion vector of the previous frame before the target frame is determined as the motion vector of the target frame; The recovered frame of the target frame is determined based on the previous frame of the target frame and the motion vector of the target frame.
[0084] In some embodiments, since H.264 allows the use of multiple reference frames, the motion vectors (MVs) of each 4×4 block in the previous coded frame may have different reference frame indices. In order to make full use of the information of the nearest neighboring frame when determining the recovered frame, the MVs of each 4×4 block of the multiple coded frames preceding the target frame are scaled to the nearest reference frame according to the temporal distance to obtain the motion vector of the frame preceding the target frame.
[0085] In one example, such as Figure 7 As shown, the keyframes for encoding are [fn-3, fn-2, fn, fn+3], and the skipped frames are [fn-1, fn+1, fn+2]. Therefore, the reference frames for fn can be fn-3 and fn-2. When estimating the MV field of the skipped frame fn+1, the MV of the reference fn-3 in fn must first be scaled to the reference fn-2. The scaling factor is determined by the inter-frame distance, i.e., (n-(n-2)) / (n-(n-3))=2 / 3.
[0086] After scaling the MV field to the nearest reference frame, if there is a skipped frame between the coded frame and its nearest reference frame, its MV field needs to be further scaled to point to its nearest preceding frame (regardless of whether the preceding frame is a coded frame or a skipped frame). If there is a skipped frame fn-1 between the coded frame fn and its nearest reference frame fn-2, when estimating the MV field of the skipped frame fn+1, the MV of the coded frame fn needs to be further scaled to point to the nearest preceding frame fn-1, with the scaling factor remaining the inter-frame distance (n-(n-1)) / (n-(n-2))=1 / 2. The scaled MV field of fn is then copied to the skipped frame fn+1, and block-based motion compensation is performed to obtain the recovered frame fn+1. Similarly, the copied MV field can be used to recover frame fn+2.
[0087] In some embodiments, since it is assumed that the target frame is skipped, the motion vector of the target frame uses the motion vector of the previous frame. The recovery frame of the target frame is determined based on the previous frame and the motion vector of the target frame.
[0088] In this embodiment, the motion vector of the previous frame of the target frame is determined as the motion vector of the target frame. Based on the previous frame of the target frame and the motion vector of the target frame, the recovery frame of the target frame is determined. Then, it is determined whether the recovery frame that the receiving end can obtain when skipping the target frame is similar to the target frame, and whether the video of the receiving end can be continuous. If it is not continuous, the target frame is determined as an encoded frame and sent to the receiving end to ensure the continuity of the video of the receiving end.
[0089] This application's embodiments reduce video encoding frame rate and bitrate by actively identifying similar frames in the video. Compared to existing rate-distortion models, it optimizes overall encoding quality by adjusting the QP (Quick Frame Rate) and determines the video skipping frame rate based on the difference between adjacent frames, thereby reducing video encoding and transmission bitrate, bandwidth usage, and video stuttering. Through layered detection, the first round of screening is performed by comparing the difference between the target frame's color histogram distribution and the color histogram distributions of adjacent frames. After detecting similar frames, a recovery frame is generated through MV (Multi-Frame Memory) copying, and a subjective quality evaluation of the region of interest (SSIM) is performed between the recovery frame and the target frame. This ensures that the receiving end can acquire frames containing important information, thereby recovering skipped frames with high quality and improving video continuity.
[0090] The embodiments of this application can be applied to surveillance scenarios. In surveillance scenarios, most surveillance video frames have similar content and scene switching is infrequent. The video transmission method provided in this application can solve the problems of high bandwidth consumption during surveillance video transmission, as well as stuttering and high latency in weak network environments. The embodiments of this application can effectively improve the quality of surveillance video transmission and reduce the network bandwidth required for wired surveillance terminal video transmission.
[0091] The embodiments of this application can be applied to screen sharing scenarios. During screen sharing, especially when sharing documents such as PowerPoint presentations, the frequency of video scene switching is low. In this scenario, using the video transmission method provided in the embodiments of this application can improve the utilization efficiency of the Central Processing Unit (CPU), reduce network transmission bandwidth, and reduce stuttering and latency.
[0092] Based on the video transmission method provided in the above embodiments, this application also provides specific implementations of a video transmission apparatus. Please refer to the following embodiments.
[0093] See Figure 8 The video transmission device 300 provided in this application embodiment includes: The acquisition module 310 is used to acquire the first color histogram distribution of the target frame and the second color histogram distribution of the frame preceding the target frame. The target frame is any frame in the video. The determination module 320 is used to determine the difference between the distribution of the first color histogram and the distribution of the second color histogram as the distribution difference of the target frame; The determination module 320 is also used to determine the recovery frame of the target frame based on the previous frame of the target frame and the motion vector of the target frame when the distribution difference of the target frame is less than a first threshold. The determination module 320 is also used to determine the structural similarity index (SSIM) between the target frame and the recovered frame; The determination module 320 is also used to determine that the target frame is a coded frame when the SSIM is less than the second threshold; The transmitting module 330 is used to transmit encoded frames to the receiving end.
[0094] In some embodiments, the device 300 may further include: The acquisition module 310 is used to acquire the distribution difference of n frames adjacent to and preceding the target frame before determining the recovery frame of the target frame based on the previous frame of the target frame and the motion vector of the target frame, where n is an integer not less than 2, when the distribution difference of the target frame is less than a first threshold. The determination module 320 is also used to determine the average value of the distribution difference of n frames as a first threshold.
[0095] In some embodiments, the device 300 may further include: The acquisition module 310 is used to acquire the SSIMs of the k coded frames adjacent to and preceding the target frame before determining that the target frame is a coded frame when the SSIM is less than the second threshold, where k is an integer not less than 2. The determination module 320 is also used to determine the average value of the SSIM corresponding to the k coded frames as a second threshold.
[0096] In some embodiments, the determining module 320 may specifically be used for: The target frame and the restored frame are divided into m regions respectively; Select the first central region from the m regions corresponding to the target frame, and select the second central region from the m regions corresponding to the recovered frame; Determine the SSIM of the first and second central regions.
[0097] In some embodiments, the device 300 may further include: The determination module 320 is also used to determine the target frame as an encoded frame when the distribution difference of the target frame is not less than a first threshold.
[0098] In some embodiments, the determining module 320 may specifically be used for: If the distribution difference of the target frame is less than the first threshold, the scaling factor for scaling the multiple encoded frames that are adjacent to and before the target frame to the frame before the target frame is determined based on the inter-frame distance between the multiple encoded frames adjacent to and before the target frame and the target frame. Determine the motion vector of the previous frame of the target frame based on the scaling factor; The motion vector of the previous frame before the target frame is determined as the motion vector of the target frame; The recovered frame of the target frame is determined based on the previous frame of the target frame and the motion vector of the target frame.
[0099] The various modules of the video transmission device provided in this application embodiment can realize the functions of each step of the video transmission method provided above, and can achieve the corresponding technical effects. For the sake of brevity, they will not be described in detail here.
[0100] Based on the same inventive concept, embodiments of this application also provide an electronic device.
[0101] Figure 9 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.
[0102] An electronic device may include a processor 401 and a memory 402 storing computer program instructions.
[0103] Specifically, the processor 401 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0104] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is non-volatile solid-state memory.
[0105] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.
[0106] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any of the video transmission methods described in the above embodiments.
[0107] In one example, the electronic device may also include a communication interface 403 and a bus 410. For example, Figure 9 As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 410 and complete communication with each other.
[0108] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0109] Bus 410 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Linear Predictive Coding (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (Peripheral Component Interconnect-X, PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application contemplates any suitable bus or interconnection. The electronic device can perform the video transmission method of the embodiments of the present invention, thereby realizing the video transmission method described above.
[0110] Furthermore, in conjunction with the video transmission methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the video transmission methods described in the above embodiments.
[0111] This application also provides a computer program product, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to perform various processes implementing any of the above-described video transmission method embodiments.
[0112] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0113] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0114] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0115] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0116] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method for transmitting video, characterized in that, include: Obtain the first color histogram distribution of the target frame and the second color histogram distribution of the frame preceding the target frame, wherein the target frame is any frame in the video; The difference between the first color histogram distribution and the second color histogram distribution is determined as the distribution difference of the target frame; If the distribution difference of the target frame is less than a first threshold, the recovery frame of the target frame is determined based on the previous frame of the target frame and the motion vector of the target frame. Determine the structural similarity index (SSIM) between the target frame and the recovered frame; If the SSIM is less than the second threshold, the target frame is determined to be an encoded frame; The encoded frame is sent to the receiving end.
2. The method according to claim 1, characterized in that, Before determining the recovered frame of the target frame based on the previous frame and the motion vector of the target frame when the distribution difference of the target frame is less than a first threshold, the method further includes: Obtain the distribution difference of n frames that are adjacent to and preceding the target frame, where n is an integer not less than 2; The average value of the distribution difference of the n frames is determined to be the first threshold.
3. The method according to claim 1, characterized in that, Before determining the target frame as a coded frame when the SSIM is less than the second threshold, the method further includes: Obtain the SSIM corresponding to the k coded frames adjacent to and preceding the target frame, where k is an integer not less than 2; The average value of the SSIM corresponding to the k encoded frames is determined to be the second threshold.
4. The method according to claim 1, characterized in that, Determining the structural similarity index (SSIM) between the target frame and the recovered frame includes: The target frame and the restored frame are each divided into m regions; A first central region is selected from the m regions corresponding to the target frame, and a second central region is selected from the m regions corresponding to the recovered frame; Determine the SSIM of the first central region and the second central region.
5. The method according to claim 1, characterized in that, The method further includes: If the distribution difference of the target frame is not less than a first threshold, the target frame is determined to be an encoded frame.
6. The method according to any one of claims 1-5, characterized in that, When the distribution difference of the target frame is less than a first threshold, determining the recovered frame of the target frame based on the previous frame of the target frame and the motion vector of the target frame includes: If the distribution difference of the target frame is less than a first threshold, a scaling factor is determined based on the inter-frame distance between the multiple encoded frames adjacent to and preceding the target frame and the target frame. The multiple encoded frames adjacent to and preceding the target frame are scaled to the frame preceding the target frame. Based on the scaling factor, determine the motion vector of the previous frame of the target frame; The motion vector of the previous frame before the target frame is determined as the motion vector of the target frame; The recovery frame of the target frame is determined based on the previous frame of the target frame and the motion vector of the target frame.
7. A video transmission device, characterized in that, include: The acquisition module is used to acquire the first color histogram distribution of the target frame and the second color histogram distribution of the frame preceding the target frame, wherein the target frame is any frame in the video; The determining module is used to determine the difference between the first color histogram distribution and the second color histogram distribution as the distribution difference of the target frame; The determining module is further configured to determine the recovery frame of the target frame based on the previous frame of the target frame and the motion vector of the target frame when the distribution difference of the target frame is less than a first threshold. The determining module is further configured to determine the structural similarity index (SSIM) between the target frame and the recovered frame; The determining module is further configured to determine the target frame as an encoded frame if the SSIM is less than the second threshold. The sending module is used to send the encoded frame to the receiving end.
8. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the video transmission method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the video transmission method as described in any one of claims 1-6.
10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the video transmission method as described in any one of claims 1-6.