Video processing method and device, electronic equipment and computer readable storage medium
Patent Information
- Application Number
- CN202310410441.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-17
- Publication Date
- 2026-10-23
- Estimated Expiration
- 2043-04-17
AI Technical Summary
[0004]本申请的目的是提供一种视频处理方法、装置、电子设备及计算机可读存储介质，以克服现有方法存在的只能利用到相邻帧或者有限的帧信息来去除帧间冗余信息的缺陷，打破已有视频压缩算法存在的瓶颈，提高视频压缩效果、降低码率
[0040]本申请实施例提供的视频处理方法，将原始视频序列编码为隐表示神经网络模型的网络模型参数；根据所述原始视频序列对所述网络模型参数进行迭代优化，直至达到预设要求为止，改善了率失真，优化了存储空间和传输带宽，克服了现有方法存在的只能利用到相邻帧或者有限的帧信息来去除帧间冗余信息的缺陷，打破了已有视频压缩算法存在的瓶颈，提高了视频压缩效果，降低了码率。
Smart Images
Figure CN116886922B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video processing technology, specifically to a video processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology
[0002] In the digital age, video data is ubiquitous in daily life. Therefore, how to efficiently store or transmit video data is a crucial issue. After decades of development, traditional hybrid video coding frameworks such as H.264 / AVC, H.265 / HEVC, and H.266 / VVC, as well as deep learning-based video coding methods, have been proposed to further improve video coding performance.
[0003] Whether it's a traditional hybrid video coding framework or a deep learning-based video coding framework, the core objective is to remove spatial and temporal redundancy from video. Although these methods are meticulously designed in various ways, either block-to-block or frame-to-frame, to remove temporal redundancy, a drawback of these existing methods is that they can only utilize adjacent frames or limited frame information to remove inter-frame redundancy. Therefore, these algorithms have performance bottlenecks, and their video compression efficiency urgently needs improvement, while bitrate needs to be reduced. Summary of the Invention
[0004] The purpose of this application is to provide a video processing method, apparatus, electronic device, and computer-readable storage medium to overcome the shortcomings of existing methods that can only utilize adjacent frames or limited frame information to remove inter-frame redundancy, break through the bottleneck of existing video compression algorithms, improve video compression effect, and reduce bitrate. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.
[0005] According to one aspect of the embodiments of this application, a video processing method is provided, including:
[0006] The original video sequence is encoded into network model parameters of a hidden representation neural network model;
[0007] The network model parameters are iteratively optimized based on the original video sequence until the preset requirements are met.
[0008] In some embodiments of this application, the iterative optimization of the network model parameters based on the original video sequence includes:
[0009] The loss function for obtaining the network model parameters is obtained based on the original video sequence;
[0010] The network model parameters are iteratively optimized based on the loss function.
[0011] In some embodiments of this application, the loss function for obtaining the network model parameters based on the original video sequence includes:
[0012] The original frames of the original video sequence are transformed into a spatial location embedding feature through convolution operations;
[0013] The reconstructed frame is obtained by using spatial location embedding features and a hidden representation neural network;
[0014] The frequency domain loss function, geometric information loss function, long-term temporal loss function, and short-term temporal loss function of the network model parameters are obtained based on the reconstructed frame and the original frame.
[0015] In some embodiments of this application, the step of obtaining the frequency domain loss function of the network model parameters based on the reconstructed frame and the original frame includes:
[0016] When monitoring the reconstructed frame, frequency domain transformation operations are used to transform the reconstructed frame and the original frame to the frequency domain space;
[0017] The difference between the reconstructed frame in the frequency domain space and the original frame in the frequency domain space is used as the frequency domain loss function.
[0018] In some embodiments of this application, obtaining the geometric information loss function of the network model parameters based on the reconstructed frame and the original frame includes:
[0019] The depth information and normal vector information of the reconstructed frame are obtained by predicting the features corresponding to the reconstructed frame, and used as the predicted values.
[0020] Predict the depth and normal vector information of the original frame as the true value;
[0021] The difference between the predicted value and the true value is obtained as the geometric information loss function.
[0022] In some embodiments of this application, obtaining long-term temporal features of the network model parameters based on the reconstructed frame and the original frame includes:
[0023] Feature information of the current reconstructed frame and past reconstructed frames is extracted, and a long-term temporal loss function is obtained by using an optimization mechanism based on Gaussian distribution prior.
[0024] In some embodiments of this application, obtaining the short-time temporal loss function of the network model parameters based on the reconstructed frame and the original frame includes:
[0025] Based on the features corresponding to the reconstructed frames, predict forward optical flow information and backward optical flow information;
[0026] Calculate the ground truth values of forward optical flow and backward optical flow information between two adjacent raw frames;
[0027] The difference between the true optical flow information and the predicted optical flow information is calculated and used as a short-time temporal loss function.
[0028] In some embodiments of this application, the iterative optimization of the network model parameters based on the original video sequence further includes:
[0029] Obtain the rate distortion of the network model parameters;
[0030] The original video sequence is divided into scenes based on the rate distortion to obtain the divided original video sequence.
[0031] The network model parameters are iteratively optimized based on the divided original video sequences until the preset requirements are met.
[0032] In some embodiments of this application, before encoding the original video sequence into network model parameters of a hidden representation neural network model, the video processing method further includes:
[0033] Based on the differences in camera scene transitions, the video to be processed is divided into scenes, resulting in multiple original video sequences.
[0034] According to another aspect of the embodiments of this application, a video processing apparatus is provided, comprising:
[0035] The encoding module is used to encode the raw video sequence into network model parameters of the hidden representation neural network model;
[0036] The optimization module is used to iteratively optimize the network model parameters based on the original video sequence until the preset requirements are met.
[0037] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video processing method described in any embodiment of this application.
[0038] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, the computer program being executed by a processor to implement the video processing method described in any embodiment of this application.
[0039] One aspect of the technical solution provided in this application embodiment may include the following beneficial effects:
[0040] The video processing method provided in this application encodes the original video sequence into network model parameters of a hidden representation neural network model; iteratively optimizes the network model parameters based on the original video sequence until a preset requirement is met, thereby improving rate distortion, optimizing storage space and transmission bandwidth, overcoming the shortcomings of existing methods that can only utilize adjacent frames or limited frame information to remove inter-frame redundancy information, breaking through the bottleneck of existing video compression algorithms, improving video compression effect, and reducing bit rate.
[0041] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 A flowchart of a video processing method according to one or more embodiments is shown.
[0044] Figure 2 A flowchart illustrating the loss function for obtaining network model parameters based on the original video sequence in one embodiment of this application is shown.
[0045] Figure 3 A flowchart of a video processing method according to one or more embodiments is shown.
[0046] Figure 4 A flowchart of a video processing method according to one or more embodiments is shown.
[0047] Figure 5 A structural block diagram of a video processing apparatus according to one or more embodiments is shown.
[0048] Figure 6 A block diagram of an electronic device structure according to one or more embodiments is shown.
[0049] Figure 7 A schematic diagram of a computer-readable storage medium according to one or more embodiments is shown. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have a meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0052] To address the issue of insufficient video compression performance in related technologies, this application provides a video processing method that encodes the original video sequence into network model parameters of a hidden representation neural network model. The network model parameters are iteratively optimized based on the original video sequence until a preset requirement is met. This improves rate distortion, optimizes storage space and transmission bandwidth, overcomes the limitations of related methods that can only utilize adjacent frames or limited frame information to remove inter-frame redundancy, breaks through the bottleneck of existing video compression algorithms, improves video compression performance, and reduces bitrate.
[0053] refer to Figure 1 As shown, one embodiment of this application provides a video processing method, which may include steps S10 to S20:
[0054] S10. Encode the original video sequence into network model parameters for the hidden representation neural network model.
[0055] For the original video sequence, a hidden representation neural network model is first used to model and represent it, thereby encoding the original video into the parameters of the neural network.
[0056] In some embodiments, the original video sequence can be divided into multiple original sub-video sequences. Each original sub-video sequence is modeled and represented using a hidden representation neural network model, thereby encoding the original sub-video sequence into the parameters of the neural network.
[0057] In some embodiments, before step S10, encoding the original video sequence into network model parameters of the hidden representation neural network model, the video processing method may further include: dividing the video to be processed into scenes according to the differences in shot scene switching, to obtain multiple original video sequences.
[0058] To fully utilize the scene change information in the video sequence, an adaptive scene segmentation mechanism is employed. This mechanism specifically includes: for a relatively long video sequence to be modeled, which may contain several different shot transition scenes, scene segmentation is performed based on these transitions. For example, for an original video sequence, the scene segmentation library function PySceneDetect is used to divide it into sub-video sequences based on the different shot transitions.
[0059] S20. Iteratively optimize the network model parameters based on the original video sequence until the preset requirements are met.
[0060] In some implementations, iteratively optimizing the network model parameters based on the original video sequence may include: obtaining a loss function for the network model parameters based on the original video sequence; and iteratively optimizing the network model parameters based on the loss function.
[0061] For example, refer to Figure 2 As shown, the loss function for obtaining the network model parameters based on the original video sequence may include: 1) transforming the original frame of the original video sequence into a spatial location embedding feature through a convolution operation; 2) reconstructing a reconstructed frame using the spatial location embedding feature and a hidden representation neural network; 3) obtaining the frequency domain loss function, geometric information loss function, long-term temporal loss function, and short-term temporal loss function for the network model parameters based on the reconstructed frame and the original frame.
[0062] For example, obtaining the frequency domain loss function of the network model parameters based on the reconstructed frame and the original frame may include: when supervising the reconstructed frame, using a frequency domain transformation operation to transform the reconstructed frame and the original frame to the frequency domain space; and using the difference between the reconstructed frame in the frequency domain space and the original frame in the frequency domain space as the frequency domain loss function.
[0063] In some embodiments, the geometric information loss function for obtaining the network model parameters based on the reconstructed frame and the original frame may include: predicting the features corresponding to the reconstructed frame to obtain the depth information and normal vector information of the reconstructed frame as predicted values; predicting the depth information and normal vector information of the original frame as true values; and obtaining the difference between the predicted values and the true values as the geometric information loss function.
[0064] For example, obtaining long-term temporal features of the network model parameters based on the reconstructed frame and the original frame may include: extracting feature information of the current reconstructed frame and past reconstructed frames, and using an optimization mechanism based on Gaussian distribution prior to obtain a long-term temporal loss function.
[0065] For example, the short-time temporal loss function for obtaining the network model parameters based on the reconstructed frame and the original frame may include: predicting forward optical flow information and backward optical flow information for the features corresponding to the reconstructed frame; calculating the ground truth values of forward optical flow information and backward optical flow information between two adjacent original frames; and calculating the difference between the ground truth optical flow information and the predicted optical flow information as the short-time temporal loss function.
[0066] In some embodiments, iteratively optimizing the network model parameters based on the original video sequence may further include: obtaining the rate-distortion of the network model parameters; dividing the original video sequence into scenes based on the rate-distortion to obtain a divided original video sequence; and iteratively optimizing the network model parameters based on the divided original video sequence until a preset requirement is met.
[0067] Since using only one network to model an entire video sequence incurs a performance loss, this application adopts a rate-distortion-based scene segmentation method. After dividing a video into several video segments using the scene segmentation method, these video segments are modeled using the neural network modeling method proposed in this application embodiment to obtain the rate-distortion under the current segmentation method. Then, based on the rate-distortion performance, the segmentation method is further improved until optimal performance is achieved.
[0068] In some embodiments, reference Figure 3 As shown, in order to make full use of the feature information provided by the original video sequence, a new intra-frame spatial location embedding mechanism and inter-frame short-term and long-term feature learning mechanism are proposed, which further improves the representation ability of the neural network model and the rate-distortion performance of video coding.
[0069] For example, firstly, a video sequence is modeled and represented using a latent representation neural network according to the video latent representation method, and then the network parameters are quantized. To further improve the modeling and representation capabilities of the latent representation neural network, the feature information of the original video sequence utilized in this embodiment mainly includes: intra-frame spatial information of the video sequence, inter-frame temporal information of the video sequence, and scene change information of the video sequence.
[0070] To fully utilize the intra-frame spatial information of the video sequence, this embodiment employs a context-dependent spatial location embedding mechanism, a frequency domain supervision mechanism, and a geometric information supervision mechanism.
[0071] Context-sensitive spatial location embedding mechanisms, specifically including:
[0072] For the original video frame v t Using a convolution operation ψ, it is transformed into a spatial location embedding feature F. e For example, the kernel size for this convolution operation can be set to 1×1, and the stride can be set to 1:
[0073] F e =(v t )
[0074] The context-related feature F e It will be used together with the hidden representation neural network to reconstruct the frames.
[0075] Frequency domain supervision mechanisms specifically include:
[0076] In the reconstructed frame During supervision, an additional frequency domain loss is added to further improve the hidden representation neural network's ability to capture high-frequency detail information. Specifically, common frequency domain transformation operations, such as Fast Fourier Transform (FFT) and Discrete Cosine Transform (DCT), are used to transform the reconstructed frames. and the original frame v t Transform to the frequency domain and then calculate the difference between them as the loss function.
[0077] Geometric information supervision mechanisms, specifically including:
[0078] For the current reconstructed frame Corresponding features This application's embodiments design an additional geometric information prediction head Ψ to predict the depth information of the current frame. and normal vector information The geometric information prediction head is a convolutional operation with a kernel size of 3×3 and a stride of 1. Subsequently, it predicts the current frame v. t depth information and normal vector information The predicted and actual values are then used as the true results. The difference between the predicted and actual values is then calculated as the loss function.
[0079] In order to make full use of the inter-frame temporal information of the video sequence, the embodiments of this application designed an optical flow field constraint mechanism to utilize the short-term temporal characteristics of the video frame, and designed a temporal comparison learning mechanism to utilize the long-term temporal characteristics of the video frame.
[0080] Optical flow field constraint mechanisms, specifically including:
[0081] In order to capture the short-term temporal information between the current reconstructed frame and its neighboring frames, for the current reconstructed frame Corresponding features This application embodiment designs an additional optical flow information prediction head φ to predict forward optical flow information. and backward optical flow information The optical flow prediction head is a convolutional operation with a kernel size of 3×3 and a stride of 1. The forward optical flow information represents the optical flow information from frame t-1 to frame t, and the backward optical flow information represents the optical flow information from frame t to frame t-1. Further, to supervise the predicted optical flow information... Using an optical flow prediction algorithm, the original frame {v t-1 ,v t Truth values of forward and backward optical flow information between} Finally, the difference between the true optical flow information and the predicted optical flow information is calculated as the loss function.
[0082] The time-series comparative learning mechanism specifically includes:
[0083] For the current reconstructed frame and previous reconstructed frame sets A contrastive learning mechanism was designed to capture long-term temporal features. Specifically, a network model was first used to extract the current reconstructed frame. And previous reconstructed frames Feature information, such as: ResNet-34 (ResNet-34 is a deep residual learning network model for image recognition), denoted as h. t and
[0084] Then, this embodiment employs an optimization mechanism based on a Gaussian distribution prior. First, for any two features u, v, this embodiment defines their cosine similarity as sim(u,v) = u T v / ||u||||v||. For the Gaussian equation, it is defined as: Where σ 2 Let represent the variance. Therefore, for frame t, the loss function is defined as:
[0085]
[0086]
[0087] Where j represents the lower table of the frame, and τ represents the temperature coefficient.
[0088] In a specific example, refer to Figure 4 As shown, a video processing method may specifically include the following steps:
[0089] Step 101: Scene division.
[0090] First, for an original video sequence, based on the different scene transitions, the scene segmentation library function PySceneDetect is used to divide it into individual sub-video sequences.
[0091] Step 102, Neural Network Encoding.
[0092] 1. For a sub-video sequence, assuming it contains a total of 80 frames, denoted as V = {v...} t ,t+1,…v t+80 Subsequently, according to the neural network encoding method proposed in the embodiments of this application, this video is encoded into neural network parameters. The specific encoding steps may include the steps of the video processing method in any of the above embodiments.
[0093] 2. During the neural network encoding stage, the loss function is calculated, and the neural network parameters are iteratively trained. In this embodiment, a mini-batch stochastic gradient descent (SGD) optimization algorithm with a batch size of 1, a momentum of 0.9, and a weight decay of 1e-5 can be used to train the neural network. The learning rate is set to 5e-4, and the maximum epoch is set to 900.
[0094] Step 103: Rate-distortion calculation.
[0095] For each scene segmentation method, a set of neural network parameters can be obtained. Each set of neural networks corresponds to a different rate-distortion. By comparing the rate-distortion of different segmentation methods, the scene segmentation method is continuously updated to achieve the best rate-distortion.
[0096] Step 104: Quantization and Decoding Process.
[0097] like Figure 4 As shown, for a video segment, this embodiment first encodes it into a set of neural network parameters through the above three steps. Then, the network parameters are quantized to further optimize storage space. Finally, the network parameters are transmitted to the decoder to obtain the reconstructed video sequence. It should be noted that the network framework of the decoding network is exactly the same as that of the encoding network. However, the network parameters of the decoding network are the quantized parameters of the encoding network.
[0098] Tests were conducted using the AVS video surveillance dataset and UVG natural scene data, compared with the default VTM12.0 configuration. The surveillance sequences included Outdoor_Crowd (Crowd), Bulong_Day_traffic (Bulong), and Night_Traffic (Night). The natural scene sequences included Bosphorus, HoneyBee, and ShakeNDry. Referring to Table 1, the method of this embodiment achieved BDBR savings of 21.98% and 13.45% compared to VTM12.0 on the surveillance sequences and natural scene sequences, respectively. The BDBR (%) performance of this embodiment significantly surpasses that of VTM12.0, further demonstrating the superiority of this embodiment.
[0099] Table 1 shows the results obtained by the method of this application embodiment on monitoring sequences and natural scene sequences.
[0100]
[0101] BDBR represents the percentage of bitrate saved by a superior encoding method under the same objective quality. The smaller the BDBR, the better the compression performance of the current encoder.
[0102] The video processing method in this application no longer utilizes only a limited number of frames to remove video redundancy. Instead, it first leverages the powerful representational capabilities of neural networks to find a continuous and compact representation space for the original video sequence. In other words, this method encodes the original video sequence into neural network parameters, which improves decoding speed and performance. Simultaneously, this method uses a neural network parameter quantization mechanism to quantize network parameters, further optimizing storage space and transmission bandwidth. By fully mining the intra-frame spatial features, inter-frame temporal features, and scene change features of the video sequence, the representational capabilities of the neural network are further enhanced.
[0103] refer to Figure 5 As shown, one embodiment of this application provides a video processing apparatus, including:
[0104] The encoding module is used to encode the raw video sequence into network model parameters of the hidden representation neural network model;
[0105] The optimization module is used to iteratively optimize the network model parameters based on the original video sequence until the preset requirements are met.
[0106] In some embodiments, the iterative optimization of the network model parameters based on the original video sequence performed by the optimization module includes: obtaining a loss function for the network model parameters based on the original video sequence; and iteratively optimizing the network model parameters based on the loss function.
[0107] In some embodiments, the loss function executed by the optimization module to obtain the network model parameters based on the original video sequence includes: transforming the original frame of the original video sequence into a spatial location embedding feature through a convolution operation; reconstructing a reconstructed frame using the spatial location embedding feature and a hidden representation neural network; and obtaining a frequency domain loss function, a geometric information loss function, a long-term temporal loss function, and a short-term temporal loss function for the network model parameters based on the reconstructed frame and the original frame.
[0108] In some embodiments, the frequency domain loss function executed by the optimization module to obtain the network model parameters based on the reconstructed frame and the original frame includes: when supervising the reconstructed frame, using a frequency domain transformation operation to transform the reconstructed frame and the original frame to the frequency domain space; and using the difference between the reconstructed frame in the frequency domain space and the original frame in the frequency domain space as the frequency domain loss function.
[0109] In some embodiments, the geometric information loss function executed by the optimization module to obtain the network model parameters based on the reconstructed frame and the original frame includes: predicting the features corresponding to the reconstructed frame to obtain the depth information and normal vector information of the reconstructed frame as predicted values; predicting the depth information and normal vector information of the original frame as true values; and obtaining the difference between the predicted values and the true values as the geometric information loss function.
[0110] In some embodiments, the optimization module performs the following steps to obtain long-term temporal features of the network model parameters based on the reconstructed frame and the original frame: extracting feature information of the current reconstructed frame and past reconstructed frames, and using an optimization mechanism based on Gaussian distribution prior to obtain a long-term temporal loss function.
[0111] In some embodiments, the short-time temporal loss function executed by the optimization module to obtain the network model parameters based on the reconstructed frame and the original frame includes: predicting forward optical flow information and backward optical flow information for the features corresponding to the reconstructed frame; calculating the ground truth values of the forward optical flow information and backward optical flow information between two adjacent original frames; and calculating the difference between the ground truth optical flow information and the predicted optical flow information as the short-time temporal loss function.
[0112] In some embodiments, the iterative optimization of the network model parameters based on the original video sequence performed by the optimization module further includes: obtaining the rate distortion of the network model parameters; dividing the original video sequence into scenes based on the rate distortion to obtain a divided original video sequence; and iteratively optimizing the network model parameters based on the divided original video sequence until a preset requirement is met.
[0113] In some embodiments, the video processing apparatus further includes a segmentation module for segmenting the video to be processed according to the differences in shot scene switching, before encoding the original video sequence into network model parameters of a hidden representation neural network model, to obtain multiple original video sequences.
[0114] The video processing apparatus provided in this application encodes the original video sequence into network model parameters of a hidden representation neural network model; iteratively optimizes the network model parameters based on the original video sequence until a preset requirement is met, thereby improving rate distortion, optimizing storage space and transmission bandwidth, overcoming the shortcomings of existing methods that can only utilize adjacent frames or limited frame information to remove inter-frame redundancy information, breaking through the bottleneck of existing video compression algorithms, improving video compression effect, and reducing bit rate.
[0115] Another embodiment of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above embodiments.
[0116] refer to Figure 6 As shown, the electronic device 10 may include: a processor 100, a memory 101, a bus 102 and a communication interface 103. The processor 100, the communication interface 103 and the memory 101 are connected through the bus 102. The memory 101 stores a computer program that can run on the processor 100. When the processor 100 runs the computer program, it executes the method provided in any of the foregoing embodiments of this application.
[0117] The memory 101 may include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 103 (which may be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.
[0118] Bus 102 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. Memory 101 is used to store programs. After receiving an execution instruction, processor 100 executes the program. The methods disclosed in any of the foregoing embodiments of this application can be applied to processor 100, or implemented by processor 100.
[0119] Processor 100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 100 or by instructions in software form. The processor 100 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), an Off-the-shelf Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 101. The processor 100 reads the information in memory 101 and, in conjunction with its hardware, completes the steps of the above method.
[0120] The electronic devices and methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.
[0121] Another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the methods described in any of the above embodiments. Reference Figure 7 As shown, the computer-readable storage medium is an optical disc 20, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the methods provided in any of the aforementioned embodiments.
[0122] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.
[0123] The computer-readable storage medium provided in the above embodiments of this application and the method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.
[0124] It should be noted that:
[0125] The term "module" is not intended to be limited to a specific physical form. Depending on the application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. Furthermore, different modules may share common components or even be implemented using the same components. Clear boundaries may or may not exist between different modules.
[0126] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used with the examples based on this. The required structure for constructing such devices is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages is for the purpose of disclosing the best mode of implementation of this application.
[0127] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0128] The above embodiments merely illustrate the implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A video processing method, characterized in that, include: The original video sequence is encoded into network model parameters of a hidden representation neural network model; The network model parameters are iteratively optimized based on the original video sequence until the preset requirements are met; The iterative optimization of the network model parameters based on the original video sequence includes: The loss function for obtaining the network model parameters is obtained based on the original video sequence; The network model parameters are iteratively optimized based on the loss function. Obtain the rate distortion of the network model parameters; The original video sequence is divided into scenes based on the rate distortion to obtain the divided original video sequence. The network model parameters are iteratively optimized based on the divided original video sequences until the preset requirements are met.
2. The method according to claim 1, characterized in that, The loss function for obtaining the network model parameters based on the original video sequence includes: The original frames of the original video sequence are transformed into a spatial location embedding feature through convolution operations; The reconstructed frame is obtained by using spatial location embedding features and a hidden representation neural network; The frequency domain loss function, geometric information loss function, long-term temporal loss function, and short-term temporal loss function of the network model parameters are obtained based on the reconstructed frame and the original frame.
3. The method according to claim 2, characterized in that, The frequency domain loss function for obtaining the network model parameters based on the reconstructed frame and the original frame includes: When monitoring the reconstructed frame, frequency domain transformation operations are used to transform the reconstructed frame and the original frame to the frequency domain space; The difference between the reconstructed frame in the frequency domain space and the original frame in the frequency domain space is used as the frequency domain loss function.
4. The method according to claim 2, characterized in that, The geometric information loss function for obtaining the network model parameters based on the reconstructed frame and the original frame includes: The depth information and normal vector information of the reconstructed frame are obtained by predicting the features corresponding to the reconstructed frame, and used as the predicted values. Predict the depth and normal vector information of the original frame as the true value; The difference between the predicted value and the true value is obtained as the geometric information loss function.
5. The method according to claim 2, characterized in that, Based on the reconstructed frame and the original frame, long-term temporal features of the network model parameters are obtained, including: Feature information of the current reconstructed frame and past reconstructed frames is extracted, and a long-term temporal loss function is obtained by using an optimization mechanism based on Gaussian distribution prior.
6. The method according to claim 2, characterized in that, The short-time temporal loss function for obtaining the network model parameters based on the reconstructed frame and the original frame includes: Based on the features corresponding to the reconstructed frames, predict forward optical flow information and backward optical flow information; Calculate the ground truth values of forward optical flow and backward optical flow information between two adjacent raw frames; The difference between the true optical flow information and the predicted optical flow information is calculated and used as a short-time temporal loss function.
7. The method according to any one of claims 1-6, characterized in that, Before encoding the original video sequence into network model parameters of a hidden representation neural network model, the video processing method further includes: Based on the differences in camera scene transitions, the video to be processed is divided into scenes, resulting in multiple original video sequences.
8. A video processing apparatus, characterized in that, include: The encoding module is used to encode the raw video sequence into network model parameters of the hidden representation neural network model; The optimization module is used to iteratively optimize the network model parameters based on the original video sequence until the preset requirements are met. The iterative optimization of the network model parameters based on the original video sequence includes: The loss function for obtaining the network model parameters is obtained based on the original video sequence; The network model parameters are iteratively optimized based on the loss function. Obtain the rate distortion of the network model parameters; The original video sequence is divided into scenes based on the rate distortion to obtain the divided original video sequence. The network model parameters are iteratively optimized based on the divided original video sequences until the preset requirements are met.
9. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video processing method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor to implement the video processing method as described in any one of claims 1-7.