Video coding parameter processing method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a video coding parameter processing method and device, a storage medium and electronic equipment, and the method comprises the steps: inputting a to-be-coded target video frame into a trained just noticeable distortion (JND) prediction model, and obtaining a predicted JND level of each video block in the target video frame; the target video frame is input into a visual attention model, the visual importance level of each video block is obtained, and the visual importance level of each video block is used for representing the visual importance degree of each video block in the target video frame; according to the predicted JND level of each video block and the visual importance level of each video block, adjusting a parameter value of a quantization parameter of each video block to obtain a target parameter value corresponding to each video block; the problem that the visual redundancy estimation accuracy is low in the prior art is solved. The visual redundancy estimation accuracy is improved, and meanwhile, the visual redundancy which is difficult to perceive is eliminated to the maximum extent.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communications, and more specifically, to a parameter processing method and device for video coding, a storage medium, and an electronic device. Background Art

[0002] The human visual system (HVS) cannot perceive image distortion below a certain threshold due to its underlying physiological and psychological mechanisms. This threshold is called the Just Noticeable Distortion (JND) threshold. In the field of image processing, the JND threshold can be used to measure the sensitivity of the human eye to distortion in different areas of an image.

[0003] In the related art, multiple JND models have been proposed, and these JND models can be mainly divided into two categories, namely, JND models based on pixel domain and JND models based on transform domain; however, the JND model based on pixel domain mainly considers brightness adaptive masking and contrast masking, while the JND model based on transform domain usually only considers the visual masking effects brought by three types of blocks: smooth blocks, edge blocks and texture blocks; the above-mentioned related technologies ignore the correlation between video blocks and video blocks, and do not consider the distortion caused by quantization of image blocks, resulting in the JND model being unable to accurately estimate the perceptual threshold applied to perceptual video coding.

[0004] It can be seen that the parameter processing method of video encoding in the related art has the problem of low accuracy in visual redundancy estimation. Summary of the invention

[0005] The embodiments of the present application provide a parameter processing method, device, computer storage medium, electronic device and computer program product for video coding, so as to at least solve the problem of low accuracy of visual redundancy estimation in the related art.

[0006] According to an embodiment of the present application, a parameter processing method for video coding is provided, including:

[0007] In an exemplary embodiment, a target video frame to be encoded is input into a trained just perceptible distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame, wherein the JND prediction model is used to predict the JND levels of the video blocks in the input video frame respectively based on a set of visual parameters corresponding to the video blocks in the input video frame;

[0008] Inputting the target video frame into a visual attention model to obtain a visual importance level of each video block, wherein the visual importance level of each video block is used to represent the degree of visual importance of each video block in the target video frame, and the visual attention model is used to predict the visual importance level of the video block in the input video frame based on the significance of the pixel points in the video block in the input video frame;

[0009] According to the predicted JND level of each video block and the visual importance level of each video block, a parameter value of a quantization parameter of each video block is adjusted to obtain a target parameter value corresponding to each video block.

[0010] According to another embodiment of the present application, a video coding parameter processing device is provided, including:

[0011] A first input unit is used to input a target video frame to be encoded into a trained just perceptible distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame, wherein the JND prediction model is used to predict the JND level of the video block in the input video frame;

[0012] A second input unit is used to input the target video frame into a visual attention model to obtain a visual importance level of each video block, wherein the visual importance level of each video block is used to represent the degree of visual importance of each video block in the target video frame, and the visual attention model is used to predict the visual importance level of the video block in the input video frame based on the significance of the pixel points in the video block in the input video frame;

[0013] The adjusting unit is used to adjust the parameter value of the quantization parameter of each video block according to the predicted JND level of each video block and the visual importance level of each video block to obtain a target parameter value corresponding to each video block.

[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0017] Through the embodiment of the present application, according to the trained just perceptible distortion JND prediction model, the JND level of the video block in the input video frame is predicted to obtain the predicted JND level of each video block in the target video frame, thereby allocating the predicted JND level to each video block to obtain the block-level JND data set, and improving the accuracy of visual redundancy estimation; the target video frame is input into the visual attention model to obtain the visual importance level of each video block, thereby, each video block with different visual importance can be accurately identified from the target video frame; and then the parameter value of the quantization parameter of each video block is adjusted according to the predicted JND level of each video block and the visual importance level of each video block, thereby, the parameter value of the different quantization parameters can be accurately allocated to each video block in the video frame according to the perceptible quality level (JND level) and the visual importance, so as to eliminate the imperceptible visual redundancy to the maximum extent, and reduce the bit rate while maintaining the visual quality. Therefore, the parameter processing method of video encoding in the related art can solve the problem of low accuracy of visual redundancy estimation, and achieve the effect of eliminating the imperceptible visual redundancy to the maximum extent while improving the accuracy of visual redundancy estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of a system architecture for optional video coding parameter processing provided in an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of an optional video encoding parameter processing flow provided in an embodiment of the present application;

[0021] Figure 4 This is a schematic diagram of an optional video frame and video block conversion provided by an embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of JND prediction model training provided in an embodiment of the present application;

[0023] Figure 6 This is an optional quantization parameter encoding schematic diagram provided in an embodiment of the present application;

[0024] Figure 7 is a structural block diagram of an image processing device based on virtual reality according to an embodiment of the present application;

[0025] Figure 8 It is a block diagram of the computer system structure of the electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0030] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0031] 1) Human Visual System (HVS), also known as the human eye visual system. People usually regard some data contained in chrominance signals, image high-frequency signals and motion images as redundant information, and call it visual redundancy. The characteristics of the human visual system can be used to compress visual redundancy, that is, to remove image data that is invisible or dispensable to the human eye.

[0032] 2) Just Noticeable Distortion (JND), also known as just noticeable distortion, is used to indicate the maximum image distortion that the human eye cannot perceive, and reflects the human eye's tolerance for image changes. The difference perception threshold refers to the minimum difference in stimuli that can just cause a difference in perception (i.e., just noticeable distortion); the ability to perceive this minimum difference is called difference sensitivity. Difference sensitivity is also inversely proportional to the difference threshold in terms of value. The smaller the difference threshold, that is, the smaller the minimum difference between stimuli that can just cause a difference in perception, the greater the difference sensitivity. In the field of image processing, the just noticeable error can be used to measure the sensitivity of the human eye to distortion in different areas of the image.

[0033] 3) Structural Similarity Index Measure (SSIM): SSIM can measure the degree of distortion of an image, or the degree of similarity between two images. SSIM is more in line with the intuitive perception of the human eye.

[0034] 4) Quantization Parameter (QP), which reflects the compression of spatial details. The smaller the value, the finer the quantization, the higher the image quality, and the longer the bit stream generated. For example, when QP is small, most details will be retained; when QP is increased, some details will be lost, the bit rate will be reduced, but the image distortion will be enhanced and the quality will be reduced.

[0035] 5) Computer Vision (CV) is a science that studies how to make machines "see". In other words, it uses cameras and computers to replace human eyes to identify and measure targets, and further processes images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data.

[0036] The embodiments of the present application provide a parameter processing method, device, computer storage medium, electronic device and computer program product for video coding, which can improve the accuracy of visual redundancy estimation while maximally eliminating imperceptible visual redundancy. The following describes an exemplary application of the terminal device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminal devices such as image acquisition devices, laptops, tablet computers, desktop computers, set-top boxes, smart TVs, vehicle terminals, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers or other types of processing devices. The following describes an exemplary application when the device is implemented as a terminal device.

[0037] refer to Figure 1 , Figure 1 The system architecture diagram of an optional video coding parameter processing provided by the embodiment of the present application is shown; a video coding parameter processing system 100 is exemplarily shown, including a terminal device 101, a network 102, and a processing device 103; the terminal device 101 can be connected to the processing device 103 through the network 102, and the network 102 can include but is not limited to at least one of the following: a wired network, a wireless network. The above-mentioned wired network can include but is not limited to at least one of the following: a wide area network, a metropolitan area network, and a local area network, and the above-mentioned wireless network can include but is not limited to at least one of the following: WIFI (Wireless Fidelity, wireless fidelity), Bluetooth.

[0038] The terminal device 101 sends the target video frame to be encoded to the processing device 103 through the network 102. The processing device 103 receives the target video frame to be encoded, and based on the trained just noticeable distortion (JND) prediction model, predicts the JND level of each video block in the input video frame to obtain the predicted JND level of each video block; the processing device 103 inputs the target video frame into the visual attention model to obtain the visual importance level of each video block; the processor 103 adjusts the quantization parameter value of each video block according to the predicted JND level of each video block and the visual importance level of each video block to obtain the target parameter value corresponding to each video block; the processing device 103 returns the target parameter value corresponding to each video block to the terminal device 101 through the network 102.

[0039] It should be noted that the trained just noticeable distortion (JND) prediction model and visual attention model can also be pre-deployed in the terminal device 101, so that the terminal device 101 can process the target video frame to be encoded independently in a scenario without an Internet connection, and accurately assign different quantization parameter values ​​to each video block in the video frame based on the joint perceptible quality level (JND level) and visual importance, so as to eliminate imperceptible visual redundancy to the greatest extent possible and reduce the bit rate while maintaining visual quality.

[0040] For example, the embodiments of the present application can be applied to scenarios such as video conferencing, video compression, and video transmission. It can greatly reduce the bit rate of perceived video encoding without losing visual quality. It can also be applied as a built-in algorithm of a video processing unit (VPU) and a video codec chip.

[0041] Optionally, in this embodiment, the processing device 103 can be an independent physical device (for example, a physical server), or a device cluster (for example, a server cluster) or a distributed system composed of multiple physical devices, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0042] refer to Figure 2 , Figure 2 is a schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application;

[0043] Optionally, in the embodiment of the present application, the parameter processing method of the video encoding may be as follows: Figure 2 The electronic device shown is executed. Figure 2 As shown, the electronic device 200 may be the terminal device 101 or the processing device 103 described above, and the electronic device 200 includes: at least one processor 201, at least one network interface 202, a bus system 203 and a memory 204. The various components in the electronic device 200 are coupled together through the bus system 203. It can be understood that the bus system 203 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 203 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 203 is not described in detail. Figure 2 Various buses are labeled as bus system 203.

[0044] The processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0045] The memory 204 may be removable, non-removable or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drive, optical disk drive, etc. The memory 204 may optionally include one or more storage devices physically away from the processor 201. The memory 204 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 204 described in the embodiments of the present application is intended to include any suitable type of memory.

[0046] In some embodiments, the memory 204 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0047] Operating system 2041, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0048] A network communication module 2402, used to reach other computing devices via one or more (wired or wireless) network interfaces 202, exemplary network interfaces 202 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB), etc.;

[0049] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The parameter processing device 2043 of video encoding stored in the memory 204 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a first input unit, a second input unit, and an adjustment unit. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0050] In some embodiments, the terminal or server can implement the parameter processing method of video encoding provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (Application, referred to as APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0051] At present, JND models are mainly divided into two categories: pixel domain JND model and transform domain JND model. The traditional pixel domain JND model mainly considers brightness adaptive masking and contrast masking; the traditional discrete cosine transform (DCT) domain JND model contrast masking mainly considers the visual masking effects brought by three types of blocks: smooth blocks, edge blocks and texture blocks;

[0052] In the related art, the JND model ignores the impact of texture video blocks of different complexity, the association between video blocks in a video frame, and the distortion caused by the quantization parameters of the video frame, resulting in low accuracy of visual redundancy estimation by the JND model in the related art.

[0053] Through the embodiments of the present application, the target video frame is input into a trained just noticeable distortion (JND) prediction model to obtain a predicted JND level for each video block in the video frame, and the target video frame is input into a visual attention model to obtain a visual importance level for each video block; the predicted JND level for each video block and the visual importance level for each video block are combined to adjust the parameter value of the quantization parameter of each video block to obtain a target parameter value corresponding to each video block; thereby, a predicted JND level and a perceptual importance are assigned to each video block, so that the accuracy of visual redundancy estimation is further improved, and the parameter value of the quantization parameter for each video block is quantitatively adjusted by combining the perceptible quality level and the visual importance, which can eliminate imperceptible visual redundancy to the maximum extent and reduce the bit rate while maintaining the visual quality.

[0054] Optionally, the parameter processing method for video coding may be executed by the terminal device 101 or the processing device 103 alone, or may be executed by the processing device 103 and a network device in the terminal device 101 together, or may be executed by other processing devices other than the processing device 103 and the terminal device 101. As an optional implementation, the parameter processing method for video coding in this embodiment is executed by the processing device 103 as an example.

[0055] refer to Figure 3 , Figure 3 is a schematic diagram of an optional video encoding parameter processing flow provided in an embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0056] In step S301, a target video frame to be encoded is input into a trained Just Noticeable Distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame.

[0057] The JND prediction model is used to predict the JND level of the video block in the input video frame.

[0058] In some embodiments, the target video frame is input into a pre-trained Just Perceptible Distortion (JND) prediction model, which predicts the JND level of each video block in the video frame based on the visual parameters corresponding to each video block in the input video frame. Here, the JND level prediction is performed for each video block in the video frame rather than the video frame to obtain the predicted JND level of each video block. This allows for accurate allocation of the predicted JND level according to the different perceptual importance of each video block, and allows for acquisition of a block-level (i.e., video block-level) JND data set to facilitate subsequent model training.

[0059] Here, a set of visual parameters of the video block is the texture complexity, prediction residual value and structural measurement index value at the video block level obtained through the information of the original video frame; for each video block in the video frame, a trained just noticeable distortion (JND) prediction model is used to assign a predicted JND level to each video block based on a set of visual parameters (visual features) corresponding to each video block.

[0060] In some embodiments, the visual parameter of the video block can be a structure similarity index measure (SSIM), which considers three key features of the video block: luminance, contrast, and structure, where luminance is measured by average grayscale, which is obtained by averaging the values ​​of all pixels; contrast is measured by grayscale standard deviation; and structure is measured by correlation coefficient.

[0061] The trained just noticeable distortion (JND) prediction model may be a pre-trained JND prediction model based on convolutional neural networks (CNN).

[0062] refer to Figure 4 , Figure 4It is an optional video frame and video block conversion schematic diagram provided in an embodiment of the present application; it includes a video frame (one video frame, the size of the video frame is 128×128) and a video block (four video blocks, the size of the video blocks are all 64×64).

[0063] It should be noted that a video frame can be a complete video image in a video sequence, or multiple continuous video blocks in a complete video image. A video frame can also be regarded as a large-sized video block composed of multiple small-sized video blocks.

[0064] Here, it is exemplified that a video frame is evenly divided into four video blocks of the same size. In actual application scenarios, a video frame can be divided into video blocks of different sizes and shapes, and the video blocks can be irregular shapes.

[0065] In some embodiments, JND is closer to the human visual system and is consistent with the subjective perception of the human eye. The JND model can be used to correct the distorted image to eliminate imperceptible errors. The predicted JND level can be a JND value. Here, the JND value ranges from 0 to 10 points. The JND value is calculated based on a 32×32 pixel block on the image screen. When the JND value of a video block is 0, the video block (which can also be a video frame or an image) is equivalent to the original video block; when the JND value of a video block is 1, an average observer cannot perceive the damage of the video block; when the JND value of a video block is 2-3, an average observer can hardly perceive the damage of the video block; when the JND value of a video block is 3-10, an average observer can perceive the damage of the video block.

[0066] Through the embodiments provided in the present application, a predicted JND level is assigned to each video block in a video frame, and based on the predicted JND level of the video block, edge and non-edge texture masking effects can be distinguished to obtain a more accurate JND estimation value.

[0067] In step S302, the target video frame is input into a visual attention model to obtain a visual importance level of each video block.

[0068] The visual importance level of each video block is used to represent the degree of visual importance of each video block in the target video frame, and the visual attention model is used to predict the visual importance level of the video block in the input video frame based on the significance of the pixel points in the video block in the input video frame.

[0069] The visual importance level of each video block in the input target video frame is predicted based on the significance of the pixels in the video block through the visual attention model to obtain the visual importance level of each video block. The visual importance level of each video block is used to represent the degree of visual importance of each video block in the target video frame. Thus, the visual importance level of each video block in the target video frame is obtained to make corresponding adjustments for video blocks of different visual importance.

[0070] In some embodiments, the visual attention model can be a Squeeze-and-Excitation Network (SENet), which is divided into two parts: compression and excitation. The purpose of the compression part is to compress the global (video frame) spatial information, and then perform feature learning in the channel dimension to form the importance of each channel. Finally, different weights are assigned to each channel through the excitation part.

[0071] In step S303, the parameter value of the quantization parameter of each video block is adjusted according to the predicted JND level of each video block and the visual importance level of each video block to obtain a target parameter value corresponding to each video block.

[0072] For each video block in a video frame, the predicted JND level of each video block and the visual importance level of each video block are obtained based on the JND prediction model and the visual attention model respectively, and the parameter value of the quantization parameter of each video block is adjusted according to the predicted JND level of each video block and the visual importance level of each video block to obtain the target parameter value corresponding to each video block. The perceptual metric and visual features can be effectively used to map the JND level from the video frame to the video block, and the parameter value of the quantization parameter of the video block can be adjusted, which can greatly reduce the bit rate of perceptual video encoding without losing visual quality.

[0073] In some embodiments, the quantization parameter reflects the compression of spatial details. The smaller the quantization parameter value, the finer the quantization, the higher the image quality, and the longer the generated bitstream. For example, when the quantization parameter value is small, the image details of the video block will be preserved, and when the quantization parameter is large, the image details of the video block are distorted and the image quality of the video block is reduced.

[0074] The quantization parameter is the process of mapping the continuous value of the signal (pixels of the video block) into multiple discrete amplitudes, realizing a many-to-one mapping of the signal value. The quantization parameter can effectively reduce the value range of the signal, thereby obtaining a better image compression effect. Quantization is the root cause of distortion; the parameter value range of the quantization parameter is 0-51. When the parameter value of the quantization parameter is 0, it indicates the finest quantization, that is, the highest image quality and the largest image bitrate. When the parameter value of the quantization parameter is 51, it indicates the coarsest quantization, that is, the lowest image quality and the smallest image bitrate.

[0075] Through the above steps of the present application, the target video frame to be encoded is input into the trained just perceptible distortion JND prediction model to obtain the predicted JND level of each video block in the target video frame, wherein the JND prediction model is used to predict the JND level of the video blocks in the input video frame based on a set of visual parameters corresponding to the video blocks in the input video frame; the target video frame is input into the visual attention model to obtain the visual importance level of each video block, wherein the visual importance level of each video block is used to indicate the degree of visual importance of each video block in the target video frame, and the visual attention model is used to predict the visual importance level of the video blocks in the input video frame based on the significance of the pixels in the video blocks in the input video frame; according to the predicted JND level of each video block and the visual importance level of each video block, the parameter value of the quantization parameter of each video block is adjusted to obtain the target parameter value corresponding to each video block. The method solves the problem of low accuracy of visual redundancy estimation in the parameter processing method of video encoding in the related art, and achieves the effect of maximally eliminating imperceptible visual redundancy while improving the accuracy of visual redundancy estimation.

[0076] In an exemplary embodiment, adjusting the parameter value of the quantization parameter of each video block according to the predicted JND level of each video block and the visual importance level of each video block to obtain a target parameter value corresponding to each video block includes:

[0077] S11, performing the following parameter value adjustment operation on each video block as the current video block to obtain a target parameter value corresponding to each video block, wherein the predicted JND level of the current video block is the i-th JND level among I JND levels, and the visual importance level of the current video block is the m-th visual importance level among M visual importance levels, wherein 1≤i≤I, 1≤m≤M;

[0078] S12, obtaining an initial quantization parameter value corresponding to the current video block, wherein the initial quantization parameter value is an initial value of the quantization parameter configured for the current video block;

[0079] S13, when the initial quantization parameter value is greater than the quantization parameter value corresponding to the current video block and the i-th JND level and is less than or equal to the quantization parameter value corresponding to the current video block and the (i+1)-th JND level, if (i+1+Mm)≤I, the quantization parameter value corresponding to the current video block and the (i+1+Mm)-th JND level is determined as the target parameter value corresponding to the current video block;

[0080] S14, when the initial quantization parameter value is greater than the quantization parameter value corresponding to the current video block and the i-th JND level and is less than or equal to the quantization parameter value corresponding to the current video block and the (i+1)-th JND level, if I<(i+1+Mm), compare the quantization parameter value corresponding to the current video block and the I-th JND level with the initial quantization parameter value, and determine the maximum value therebetween as the target parameter value corresponding to the current video block.

[0081] In some embodiments, the predicted JND level of the current video block is determined to be the i-th JND level among I JND levels, and the visual importance level of the current video block is the m-th visual importance level among M visual importance levels; each JND level corresponds to a quantization parameter value.

[0082] An initial quantization parameter value corresponding to the current video block is obtained. When the initial quantization parameter value is greater than the quantization parameter value corresponding to the i-th JND level and less than or equal to the quantization parameter value corresponding to the (i+1)-th JND level, if (i+1+Mm)≤I, the quantization parameter value corresponding to the (i+1+Mm)-th JND level is used as the target parameter value corresponding to the current video block. If I<the quantization parameter value corresponding to the (i+1+Mm)-th JND level, the maximum value between the quantization parameter value corresponding to the I-th JND level and the initial quantization parameter value is used as the target parameter value corresponding to the current video block.

[0083] It can be represented by the following quantitative control algorithm (program code):

[0084]

[0085]

[0086] Among them, B b Indicates the current video block b, QP t Indicates the initial quantization parameter of the current video block, JND b，i Indicates that the JND level of the current video block is level i, 1≤i≤I, and i is an integer, VI b Indicates the visual importance of the current video block, m is used to indicate the visual importance level, 1≤m≤M, and m is an integer, QP b，outputrepresents the target quantization parameter value corresponding to the current video block, that is, the maximum value between the quantization parameter value corresponding to the I-th JND level and the initial quantization parameter value.

[0087] Here, according to the definition of JND, between two adjacent JND levels, encoding any quantization parameter for the current video block will not cause a difference in visual perceptual quality. b =M), based on the quantization parameter assigned by the quantization control algorithm of the video block, that is, the upper boundary of the target quantization parameter (the maximum value of the quantization parameter value corresponding to the I-th JND level and the initial quantization parameter value), the video block can show the perceptual quality corresponding to the quantization parameter to improve the visual perceptual quality of the important video block. For the video blocks with lower visual importance, the parameter value of the quantization parameter corresponding to the next JND level will be assigned to reduce the bit rate.

[0088] In some embodiments, a video importance threshold may be preset to determine video blocks of lower visual importance. The video importance threshold may be set according to a visual importance level.

[0089] Through the embodiments provided by the present application, the quantization parameters are adjusted for each video block in a video frame according to the characteristics of the minimum perceptible error, so as to enhance the quantization parameters of important video blocks to improve the visual quality and reduce the quantization parameters of video blocks with lower visual importance to reduce the bit rate of video encoding.

[0090] In an exemplary embodiment, after obtaining the initial quantization parameter value corresponding to the current video block, the method further includes:

[0091] S21, when the initial quantization parameter value is less than or equal to the quantization parameter value corresponding to the current video block and the first JND level, determine the quantization parameter value corresponding to the current video block and the first JND level as the target parameter value corresponding to the current video block.

[0092] In some embodiments, when the initialization quantization parameter of the video block is less than or equal to the quantization parameter value corresponding to the first JND level, the quantization parameter value corresponding to the first JND level is used as the target quantization parameter value of the video block, which can be represented by the following quantization control algorithm (program code):

[0093] if QP t <=JND b，1

[0094] QP b，output =JND b，1

[0095] end if

[0096] Among them, QP tIndicates the initial quantization parameter of the current video block, JND b，i Indicates the JND level of the current video block, 1≤i≤I, and i is an integer, QP b，output Indicates the target quantization parameter value corresponding to the current video block.

[0097] In some embodiments, the initial quantization parameter value that is less than or equal to the quantization parameter value corresponding to the current video block and the first JND level may be modulated to a quantization parameter value corresponding to another JND level to enhance the visual quality of the video block. Here, there is a mapping relationship between the JND level and the quantization parameter value.

[0098] In some embodiments, when the initial quantization parameter of the quantization parameter of the video block b is less than or equal to the quantization parameter value corresponding to the first JND level, the quantization parameter value corresponding to the first JND level is configured as the target parameter value of the video block b; when the target parameter value of the quantization parameter of the video block b is greater than or equal to the quantization parameter value corresponding to the i-th JND level and the initial quantization parameter of the quantization parameter of the video block b is less than or equal to the quantization parameter value corresponding to the i+1-th JND level, the quantization parameter value corresponding to the i+1-th JND level is configured as the target parameter value of the video block b; which can be expressed by the following formula (1):

[0099]

[0100] Among them, QP b,JND represents the target parameter value of the JND model for the video block b, JND b,1 Indicates that the video block b is the quantization parameter value corresponding to the first JND level, QP t Indicates the initial quantization parameter of the current video block, JND b,i+1 Indicates that the video block b is the quantization parameter value corresponding to the i+1th JND level, JND b,i Indicates the quantization parameter value corresponding to the i-th JND level of video block b.

[0101] Through the embodiments provided in the present application, video blocks with initial quantization parameter values ​​that are too low (lower than the quantization parameter values ​​corresponding to the first JND level) can be modulated to be the same as the quantization parameter values ​​corresponding to the first JND level, so as to avoid the visual quality of some video blocks being too low, resulting in unclear overall viewing experience.

[0102] In an exemplary embodiment, adjusting the parameter value of the quantization parameter of each video block according to the predicted JND level of each video block and the visual importance level of each video block to obtain a target parameter value corresponding to each video block includes:

[0103] S31, obtaining a quantization parameter value corresponding to each video block.

[0104] The quantization parameter value corresponding to each video block is an initial value of the quantization parameter configured for each video block.

[0105] S32: Determine an offset value of a quantization parameter corresponding to each video block according to the visual importance level of each video block.

[0106] The offset value of the quantization parameter corresponding to a video block is negatively correlated with the visual importance level of the video block.

[0107] S33: Determine the sum of the quantization parameter value corresponding to each video block and the offset value of the quantization parameter corresponding to each video block as the target parameter value corresponding to each video block.

[0108] In some embodiments, an initial value of a quantization parameter of each video block is obtained, a visual importance level of each video block is determined, and an offset value of the quantization parameter is added to a video block whose visual importance level is lower than a preset threshold (the lower the visual importance level, the higher the added quantization parameter offset value); and the sum of the initial value of the quantization parameter of each video block and the offset value of the quantization parameter of each video block is used as the target parameter value corresponding to each video block.

[0109] Here, the offset value of the quantization parameter is added to the initial value of the quantization parameter of the video block in the area with lower visual importance to increase the target parameter value of the video block in the area with lower video importance, which can reduce the image coding length without reducing the visual effect and reduce unnecessary information in the vision.

[0110] In some embodiments, the offset value of the quantization parameter configured for each video block is configured according to the visual importance level of each video block. For example, in an area where the visual importance level is 1, the offset value of the quantization parameter configured for each video block is 4; in an area where the visual importance level is 2, the offset value of the quantization parameter configured for each video block is 2; in an area where the visual importance level is 3, the offset value of the quantization parameter configured for each video block is not configured, which can be expressed by the following formula (2):

[0111]

[0112] Among them, QP b,Saliency represents the target parameter value of video block b, QP t Indicates the initial value of the quantization parameter configured for the video block (i.e., the initial quantization parameter value), VI b Indicates the visual importance level of the quantization parameter.

[0113] In some embodiments, the quantization parameter value corresponding to each video block may also be determined according to the predicted JND level of each video block and the visual importance level of each video block.

[0114] Through the embodiments provided in the present application, for video blocks in areas with lower visual importance levels, offset values ​​of quantization parameters are configured to increase target parameter values ​​of video blocks in areas with lower visual importance levels, thereby reducing image coding length and reducing unnecessary visual information without reducing visual effects.

[0115] In an exemplary embodiment, a target video frame to be encoded is input into a trained just perceptible distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame, including:

[0116] S41, inputting the target video frame to be encoded and each video block in the target video frame into the trained JND prediction model respectively, and obtaining the predicted JND level of each video block output by the JND prediction model.

[0117] refer to Figure 5 , Figure 5 It is a schematic diagram of the JND prediction model training provided in an embodiment of the present application; the JND prediction model can be a JND prediction model based on a convolutional neural network, including 6 convolutional layers, a cascade layer, a first fully connected layer, a long short-term memory network and a second fully connected layer.

[0118] In some embodiments, the trained JND prediction model can be a JND prediction model based on a convolutional neural network, and the spatial feature extraction part is input into a 3-channel 128×128 image frame and a 3-channel 64×64 image block, wherein the spatial feature extraction part represents the spatial perception information (Spatial perceptioninformation) and frequency domain perception information (Frequency domain perception information) of the video block (video frame), and the visual distortion perceived by the human visual system can be determined based on the spatial feature extraction part to obtain the predicted JND level of each video block output by the JND prediction model.

[0119] Through the embodiments provided in the present application, the target video frame to be encoded and each video block in the target video frame can be respectively input into the trained JND prediction model to directly obtain the predicted JND level of each video block.

[0120] In an exemplary embodiment, a target video frame to be encoded is input into a trained just perceptible distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame, including:

[0121] S51, input the target video frame and each video block into a set of convolutional layers of the trained JND prediction model, and fuse the features extracted by the last convolutional layer in the set of convolutional layers into spatial features corresponding to each video block through the cascade layer of the JND prediction model.

[0122] S52, input each video block into the frequency domain feature extraction layer of the JND prediction model, and fuse the frequency domain perception features extracted by the frequency domain feature extraction layer into energy features corresponding to each video block through a cascade layer.

[0123] S53, inputting the spatial features corresponding to each video block and the energy features corresponding to each video block into the first fully connected layer, the long short-term memory network, and the second fully connected layer connected in series in the JND prediction model to obtain the predicted JND level of each video block output by the JND prediction model.

[0124] refer to Figure 5 , Figure 5 It is a schematic diagram of the JND prediction model training provided in an embodiment of the present application; the JND prediction model can be a JND prediction model based on a convolutional neural network, including a group of convolutional layers (6 convolutional layers), a frequency domain feature extraction layer (refer to Eq-(4) in the figure), a cascade layer, a first fully connected layer, a long short-term memory network and a second fully connected layer.

[0125] In some examples, a video frame with an input size of 128×128 and 3 channels is input to the first convolutional layer (CONV_1); a video block with an input size of 64×64 and 3 channels is input to the second convolutional layer (CONV_2), where six convolutional layers are connected in series in sequence; the input of the first convolutional layer is the current training video frame, the output of the first convolutional layer passes through an average pooling layer (Me_P) and is input into the second convolutional layer in a group of convolutional layers together with the current training video block, the output of the second convolutional layer and the convolutional layer after the second convolutional layer passes through the maximum pooling layer (Ma_P) and is input into the connected convolutional layers, and the features extracted by the last convolutional layer in a group of convolutional layers are cascaded into spatial features corresponding to the current training video block through the cascade layer of the current JND prediction model.

[0126] The current training video block (size 64×64, number of channels 3, in an 8-frame video frame window) is input into the frequency domain feature extraction layer of the current JND prediction model (refer to Eq-(4) in the figure), and the frequency domain perception features extracted by the frequency domain feature extraction layer are concatenated into energy features corresponding to the current training video block through the cascade layer.

[0127] The spatial features and energy features corresponding to the current video block are sequentially input into the first fully connected layer, the long short-term memory network, and the second fully connected layer connected in series with the current JND prediction model to obtain the predicted JND level of each video block output by the current JND prediction model.

[0128] In some embodiments, the cascade layer is used to combine multiple features, for example, by fusing features extracted by multiple convolutional feature extraction frameworks or fusing multiple information of the output layer through the cascade layer to obtain an output result after combining multiple features.

[0129] In some embodiments, the JND level of the video block in the original video is determined by mapping according to the spatial domain (spatial perception information), structural similarity, and frequency domain (frequency domain perception information) of the original video, and then the video block of the original video and the JND level corresponding to the video block are input into a JND prediction model for training to obtain a trained JND prediction model; the test video sequence is sent to the trained JND prediction model to obtain the predicted JND level corresponding to each video block, and the predicted JND level is a specific level (for example, the JND level can be expressed as JND2, JND3), and the characteristic performance that conforms to the JND is within the range of distortion threshold that can be just perceived by the human eye.

[0130] Through the embodiments provided by the present application, firstly, fitting is performed after the cascade layer based on the first fully connected layer to reduce the loss of feature information; secondly, a special recurrent neural network that can capture long-term dependencies is used through a long short-term memory network to avoid long-term dependency problems; finally, the highly abstract features that have undergone multiple convolutions can be integrated through the second fully connected layer, and then normalized, and a probability can be output for various classification situations (that is, the predicted JND level is output for each video block).

[0131] In an exemplary embodiment, the target video frame and each video block are input into a set of convolutional layers of a trained JND prediction model, including:

[0132] S61, input the target video frame into the first convolution layer in a group of convolution layers, and input the output of the first convolution layer into the second convolution layer in a group of convolution layers together with each video block after passing through an average pooling layer, to obtain the features extracted by the last convolution layer.

[0133] Among them, the convolutional layers in a group of convolutional layers are connected in series in sequence. The outputs of the second convolutional layer and each subsequent convolutional layer need to pass through a maximum pooling layer and then be input into the connected convolutional layer. The output of the last convolutional layer passes through a maximum pooling layer and then is input into the cascade layer.

[0134] In some embodiments, reference Figure 5The convolution (kernel) size of each convolution layer is 3×3. The output size of the first convolution layer is 128×128, and the number of channels is 16; the output size of the second convolution layer (CONV_2) is 64×64, and the number of channels is 32; the output size of the third convolution layer (CONV_3) is 32×32, and the number of channels is 64; the output size of the fourth convolution layer (CONV_4) is 16×16, and the number of channels is 128; the output size of the fifth convolution layer (CONV_5) is 8×8, and the number of channels is 256; the output size of the sixth convolution layer (CONV_6) is 4×4, and the number of channels is 512.

[0135] In some embodiments, the JND prediction model further includes an average pooling layer (Me_P) and multiple maximum pooling layers (Ma_P), referring to Figure 5 The kernel size of the average pooling layer between the first and second convolutional layers is 2×2, and the output size is 64×64; the kernel size of the maximum pooling layer between the second and third convolutional layers is 2×2, and the output size is 32×32; the kernel size of the maximum pooling layer between the third and fourth convolutional layers is 2×2, and the output size is 16×16; the kernel size of the maximum pooling layer between the fourth and fifth convolutional layers is 2×2, and the output size is 8×8; the kernel size of the maximum pooling layer between the fifth and sixth convolutional layers is 2×2, and the output size is 4×4; the kernel size of the maximum pooling layer between the sixth convolutional layer and the cascade layer is 2×2, and the output size is 2×2.

[0136] In some embodiments, the JND prediction model can be a JND prediction model based on a convolutional neural network. First, in the spatial perception information extraction part, a 3-channel 128×128 video block and a 3-channel 64×64 video block are input, and then six convolution (CONV) layers are passed, each CONV layer uses a 3×3 pixel kernel and a rectified linear unit (ReLU) function activation, an average pooling layer with a 2×2 pixel kernel is used after the first convolution layer, and a maximum pooling layer with a 2×2 pixel kernel is used after the remaining convolution layers. In order to consider the spatial domain (spatial features), frequency domain (frequency features), and time domain (temporal features) of the video block without adding too much complexity, the embodiment of the present application adds a cascade layer to the JND prediction model, which can cascade the feature information extracted from the convolution layer into spatial features; in the frequency domain perceptual information feature extraction part, a 3-channel 64×64 video block is input, and the frequency domain (perceptual) features are extracted through the frequency domain feature extraction layer (Eq-(4)) of the JND prediction model, and then the frequency features are concatenated into time features (which can also be energy features, because time features have the characteristic of concentrated energy) through the cascade layer. Then, two fully connected layers (FC layers for short) and a long short-term memory recursive neural network (LSTM) module are used to predict the JND level based on spatial features and time features (energy features).

[0137] Through the embodiments provided by the present application, an average pooling layer and multiple maximum pooling layers are added to a group of convolutional layers. The pooling process is to evenly divide the input image into several rectangular areas. The average pooling layer can take the average value of the image in the pooling area, and the obtained feature information is more sensitive to background information; the maximum pooling layer can take the maximum value of the pixel points in the pooling area, and the obtained feature information is more sensitive to texture feature information.

[0138] In an exemplary embodiment, before inputting the target video frame to be encoded into the trained just perceptible distortion (JND) prediction model, the method further includes:

[0139] S71, obtaining a training video sequence and annotation information corresponding to the training video sequence.

[0140] The training video sequence includes K training video frames, each of the K training video frames is divided into L training video blocks in the same division method, and the annotation information corresponding to the training video sequence includes a JND level identifier of each training video block of each training video frame, and the JND level identifier of each training video block of each training video frame is used to identify the JND level of each training video block of each training video frame, and K and L are both positive integers greater than or equal to 2;

[0141] S72, performing multiple rounds of model training on the initial JND prediction model using the training video sequence and the annotation information corresponding to the training video sequence to obtain a trained JND prediction model.

[0142] In some embodiments, a training video sequence and corresponding annotation information are obtained, the training video sequence is divided into K training video frames, each training video frame is divided into L training video blocks, K and L are both positive integers greater than or equal to 2; the specific values ​​of K and L can be determined according to the characteristics of the training video sequence (for example, video color characteristics, video image texture characteristics), and the annotation information of the training video sequence includes a JND level identifier of each video block of each training video frame, which is used to identify the JND level of each training video block of each training video frame.

[0143] In some embodiments, a training loss value is determined based on the label of the training video sequence and the real annotation information corresponding to the training video sequence, and the current neural network model is iteratively trained according to the training loss value and the training video sequence until the training loss value meets the training end condition, and the model at the end of the training is determined as the trained JND prediction model;

[0144] In some embodiments, the annotation information corresponding to the training video sequence may also be the visual importance level or the quantization parameter value of the training video sequence.

[0145] Through the embodiments provided in the present application, the initial JND prediction model can be pre-trained based on the training video sequence and the corresponding annotation information to obtain a trained JND prediction model to enhance the accuracy of the predicted JND level for each video block.

[0146] In an exemplary embodiment, multiple rounds of model training are performed on the initial JND prediction model using the training video sequence and the annotation information corresponding to the training video sequence to obtain a trained JND prediction model, including:

[0147] S81, during a round of model training, each training video block of each training video frame is sequentially used as a current training video block of a current training video frame to perform the following parameter adjustment operation.

[0148] The JND level identified by the JND level identifier of the current training video block is the first JND level, and the loss function of the JND prediction model is a preset loss function.

[0149] S82, input the current training video frame and the current training video block into the current JND prediction model to obtain the level prediction result output by the current JND prediction model.

[0150] The level prediction result is used to indicate the second JND level of the current training video block predicted by the current JND prediction model.

[0151] S83, determining function values ​​of a preset loss function corresponding to the first JND level and the second JND level, and obtaining a first function value.

[0152] S84, adjusting the model parameters of the current JND prediction model based on the first function value to obtain an adjusted current JND prediction model.

[0153] The JND level of the current training video block predicted by the adjusted current JND prediction model is the third JND level, and the first function value is greater than the function value of the preset loss function corresponding to the first JND level and the third JND level.

[0154] In some embodiments, the JND prediction model is based on a convolutional neural network model, and a Transformer model, a GAN model, an improved CNN model or other network structures may also be used to implement the prediction of the block-level JND level.

[0155] Here, the preset loss function is used to evaluate the degree of difference between the predicted value and the true value of the model. It is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. In applications, the loss function is usually associated with the optimization problem as a learning criterion, that is, the model is solved and evaluated by minimizing the loss function, for example, it is used for model parameter estimation (ParametricEstimation) in statistics and machine learning.

[0156] Through the embodiments provided in the present application, the JND prediction model can be adjusted and updated based on the loss function to accurately output the JND level prediction result.

[0157] In an exemplary embodiment, before obtaining the training video sequence and the annotation information corresponding to the training video sequence, the method further includes:

[0158] S91, obtaining spatial perception information of each training video block of the training video sequence.

[0159] The spatial perception information of each training video block is used to represent the texture complexity of each training video block.

[0160] S92: Obtain frequency domain perception information of each training video block in the training video sequence.

[0161] The frequency domain perception information of each training video block is used to represent the frequency domain energy of each training video block.

[0162] S93: Determine a perceptual parameter value of each training video block according to the spatial perceptual information of each training video block and the frequency domain perceptual information of each training video block.

[0163] The perceptual parameter value of each training video block is used to represent the tolerance of each training video block to distortion.

[0164] S94, performing the following parameter value determination operation on each training video block as a current training video block to obtain a quantization parameter value corresponding to each training video block and each JND level in a set of JND levels.

[0165] S95: Determine a distortion value corresponding to each quantization parameter value within the quantization parameter value range and the current training video block.

[0166] The distortion value corresponding to each quantization parameter value of the current training video block is determined based on structural similarity and is obtained by encoding the current training video block using each quantization parameter value, and is a distortion of the encoded video block relative to the current training video block.

[0167] S96: Determine the product of the distortion value corresponding to the current training video block and each quantization parameter value and the current perception coefficient as the perception degradation metric value corresponding to the current training video block and each quantization parameter value.

[0168] The current perceptual coefficient is the product of the perceptual parameter value of the current training video block and the first coefficient plus the second coefficient.

[0169] S97, determining the maximum distortion value corresponding to each JND level of the training video sequence.

[0170] The maximum distortion value corresponding to each JND level is the maximum value of the distortion values ​​corresponding to each training video block of the training video sequence and the preset quantization parameter value of each JND level.

[0171] S98, determining a quantization parameter value corresponding to each JND level of the current training video block according to the perceptual degradation metric value corresponding to each quantization parameter value of the current training video block and the maximum distortion value corresponding to each JND level.

[0172] The quantization parameter value corresponding to each JND level of the current training video block is the maximum value among the quantization parameter values ​​whose corresponding perceptual degradation metric value is less than or equal to the maximum distortion value corresponding to each JND level.

[0173] In some embodiments, the spatial perception information and the frequency domain perception information are used to determine the visual distortion information perceived by the human visual system. The spatial perception information (i.e., spatial complexity) of each training video block can be expressed by formula (3):

[0174] SPI b =max n {σ(scharr(B b，f ))}f∈{1,...,n} (3)

[0175] Among them, SPI b Used to represent the spatial perception information of the current training video block b, max n {σ(scharr(B b，f ))} represents the standard deviation of all pixels in each Scharr operator (filter) and the maximum value in n consecutive frames, and scharr represents the application of the Scharr operator to the current training video block b in the video frame f.

[0176] In order to map the JND level from the video frame level to the video block level, it is necessary to determine the spatial perceptual information complexity level (i.e., spatial complexity level) of the current training video block relative to the spatial perceptual information complexity level of all blocks in the frame. To this end, the spatial complexity of each video block in the video frame is converted to a value range of 0-1 through minimum and maximum normalization. The normalized spatial complexity calculation formula (4) for each video block is as follows:

[0177]

[0178] Among them, NSC b represents the normalized spatial complexity of the current training video block b (the bth video block), SPI max and SPI min They represent the maximum and minimum values ​​of the spatial perception information of the current training video block b in a video frame, SPI b Used to represent the spatial perception information of the current training video block b.

[0179] Here, the higher the normalized spatial complexity of a video block, the more complex the texture of the video block is, and the more distortion can be tolerated.

[0180] In some embodiments, the frequency domain perception information (ie, frequency domain energy value) of each video block can be expressed by the following formulas (5)-(6):

[0181] Hadamard b，k =H(B b，k -B b，k-1 )H (5)

[0182] FPT b =max k {σ(Hadamard b，k )}, k∈{1,...,K-1} (6)

[0183] Among them, Hadamard b，f represents the Hadamard matrix (also called Hadamard product) of the video block b and the video frame k, H is the preset normalized Hadamard matrix, (B b，k -B b，k-1 ) is the difference between the current training video block b of the kth training video frame and the current training video block b of the (k-1)th training video frame, FPT b Represents the frequency domain perception information of the current training video block b; max k {σ(Hadamard b，k )} represents the maximum value of the standard deviation corresponding to each Hadamard matrix in the Hadamard matrices of the video block b and the video frame k.

[0184] In some embodiments, the normalized frequency domain energy value of each video block can be expressed by the following formula (7):

[0185]

[0186] Among them, NFC b Represents the normalized frequency domain energy value of the current training video block b (the bth video block), FPT b Indicates the frequency domain energy value of the current training video block b, FPT max and FPT min They respectively represent the maximum and minimum values ​​of the frequency domain energy values ​​of the training video blocks in the training video.

[0187] According to the normalized spatial complexity (spatial perception information) of each training video block and the normalized frequency domain energy value (frequency domain perception information) of each training video block, the perception parameter value of each training video block is determined, which can be expressed by the following formula (8):

[0188]

[0189] Among them, PI b represents the perceptual parameter value (i.e., perceptual information) of the current training video block b, NFC b Represents the normalized frequency domain energy value of the current training video block b (the bth video block), NSC bNormalized spatial complexity of the current training video block b (the b-th video block).

[0190] Here, the perceptual parameter value of each training video block is used to represent the tolerance of each training video block to distortion.

[0191] Each training video block is used as a current training video block to perform the following parameter value determination operations 1 to 4 to obtain a quantization parameter value corresponding to each training video block and each JND level in a set of JND levels:

[0192] Operation 1, determining the distortion value corresponding to each quantization parameter value within the quantization parameter value range of the current training video block. For example, the distortion value corresponding to each quantization parameter value within the quantization parameter value range of the current training video block b can be obtained by the formula express.

[0193] in, It represents the structure similarity index measure (SSIM) score of the training video block b encoded with the quantization parameter, and the quantization parameter value range is the quantization parameter value range of each standard protocol (for example, the quantization parameter value range is usually 1-51).

[0194] It should be noted that the distortion value corresponding to the current training video block and each quantization parameter value is determined based on structural similarity; the distortion of the encoded video block obtained by encoding the current training video block using each quantization parameter value relative to the current training video block.

[0195] Operation 2: multiplying the distortion value corresponding to the current training video block and each quantization parameter value by the current perceptual coefficient to determine the perceptual degradation metric value corresponding to the current training video block and each quantization parameter value. The perceptual degradation metric value corresponding to the current training video block and each quantization parameter value can be expressed by the following formula (9):

[0196]

[0197] in, represents the perceptual degradation metric value corresponding to the current training video block b and each quantization parameter value qp, α represents the first coefficient, PI b represents the perceptual parameter value of the current training video block b, β represents the second coefficient, It represents the distortion value corresponding to each quantization parameter value (ie, qp) of the current training video block b.

[0198] Here, the first coefficient and the second coefficient are pre-set and can be determined according to the actual application scenario, and this application does not limit them here.

[0199] Operation 3, determining the maximum distortion value corresponding to each JND level of the training video sequence, can be expressed by the following formula (10):

[0200]

[0201] Among them, MaxD i represents the maximum distortion value corresponding to the i-th JND level, represents the SSIM score of the current training video block b and the quantization parameter pq value corresponding to the i-th JND level;

[0202] Here, the maximum distortion value corresponding to each JND level is the maximum value among the distortion values ​​corresponding to each training video block of the training video sequence and the preset quantization parameter value of each JND level;

[0203] Operation 4, determining the quantization parameter value corresponding to the current training video block and each JND level according to the perceptual degradation metric value corresponding to each quantization parameter value of the current training video block and the maximum distortion value corresponding to each JND level, which can be expressed by the following formula (11):

[0204]

[0205] Among them, JND b,i represents the quantization parameter value corresponding to the i-th JND level of the current training video block b, max(qp) represents the maximum distortion value of the quantization parameter value qp corresponding to each JND level, represents the perceptual degradation metric value corresponding to the current training video block b and each quantization parameter value qp, MaxD i Indicates the maximum distortion value corresponding to the i-th JND level.

[0206] Here, the quantization parameter value corresponding to each JND level of the current training video block is the maximum value among the quantization parameter values ​​whose corresponding perceptual degradation metric value is less than or equal to the maximum distortion value corresponding to each JND level.

[0207] It should be noted that, in addition to the embodiments provided in the present application which use block-level texture complexity, prediction residual value and SSIM value, the Scharr filter kernel for detecting texture complexity can also use the Sobel filter kernel and the angle-based edge detection filter kernel; the prediction residual value for evaluating the degree of motion can also use the frame difference method, the three-frame difference method, and the neural network-based training model, the Hadamard transform can also be replaced by the DCT transform, and the SSIM value can also use the normalized peak signal-to-noise ratio and the normalized learning-perceived image block familiarity.

[0208] Through the embodiments provided by the present application, according to formulas (9)-(11), each training video block is encoded by multiple quantization parameters to obtain the information required for JND mapping. Since this process is only performed once in training, no overhead is added during operation. When mapping and training are performed on CNN, the JND prediction of the new video block does not need to be encoded using different quantization parameters, which greatly reduces the amount of calculation.

[0209] In an exemplary embodiment, determining the perceptual parameter value of each training video block according to the spatial perceptual information of each training video block and the frequency domain perceptual information of each training video block includes:

[0210] S101, normalizing the spatial perception information of each training video block to obtain the normalized spatial complexity of each training video block.

[0211] S102, normalizing the frequency domain perception information of each training video block to obtain a normalized frequency domain energy value of each training video block.

[0212] S103: Determine the ratio of the normalized frequency domain energy value of each training video block to the normalized spatial complexity of each training video block as a perceptual parameter value of each training video block.

[0213] In order to map the JND level from the video frame level to the video block level, it is necessary to determine the spatial perceptual information complexity level (i.e., spatial complexity level) of the current training video block relative to the spatial perceptual information complexity level of all blocks in the frame. To this end, the spatial complexity of each video block in the video frame is converted to a value range of 0-1 through minimum-maximum normalization. The normalized spatial complexity calculation formula (12) for each video block is as follows:

[0214]

[0215] Among them, NSC b The normalized spatial complexity of the current training video block b (the bth video block), SPI max and SPI min They represent the maximum and minimum values ​​of the spatial perception information of the current training video block b in one frame, SPI b Used to represent the spatial perception information of the current training video block b.

[0216] Here, the higher the normalized spatial complexity of a video block, the more complex the texture of the video block is, and the more distortion can be tolerated.

[0217] According to the normalized spatial complexity (spatial perception information) of each training video block and the normalized frequency domain energy value (frequency domain perception information) of each training video block, the perception parameter value of each training video block is determined, which can be expressed by the following formula (13):

[0218]

[0219] Among them, PI b represents the perceptual parameter value (i.e., perceptual information) of the current training video block b, NFC b Represents the normalized frequency domain energy value of the current training video block b (the bth video block), NSC b Normalized spatial complexity of the current training video block b (the b-th video block).

[0220] Here, the perceptual parameter value of each training video block is used to represent the tolerance of each training video block to distortion.

[0221] In an exemplary embodiment, obtaining spatial perception information of each training video block of a training video sequence includes:

[0222] S111, performing the following acquisition operations on each training video block of the training video sequence as a current training video block to obtain spatial perception information of each training video block.

[0223] S112, applying Scharr filters to the current training video blocks of the K training video frames for filtering respectively, to obtain K Scharr filter blocks.

[0224] S113, determining the maximum value of the standard deviation corresponding to each Scharr filter block in the K Scharr filter blocks as the spatial perception information of the current training video block.

[0225] Wherein, the standard deviation corresponding to each Scharr filter block is the standard deviation of all pixels in each Scharr filter block;

[0226] In some embodiments, the number of training video frames and Scharr filter blocks is the same, that is, both are K (K is the total number of training video frames), and the spatial perception information (that is, spatial complexity) of each training video block can be expressed by formula (14):

[0227] SPI b =max k {σ(scharr(B b，k ))}k∈{1,...,K} (14)

[0228] Among them, SPI bUsed to represent the spatial perception information of the current training video block b, max k {σ(scharr(B b，k ))} represents the standard deviation of all pixels in each Scharr operator (filter) and the maximum value in K consecutive frames, scharr(B b，k ) represents applying the Scharr operator to the current training video block b in video frame k (the kth video frame).

[0229] In an exemplary embodiment, obtaining frequency domain perception information of each training video block of a training video sequence includes:

[0230] S114, performing the following acquisition operation on each training video block of the training video sequence as a current training video block to obtain frequency domain perception information of each training video block.

[0231] S115 , respectively calculating the Hadamard matrices corresponding to the respective training video frames from the second training video frame to the Kth training video frame among the K training video frames, to obtain (K−1) Hadamard matrices.

[0232] Among them, the Hadamard matrix corresponding to the kth training video frame is H×the prediction residual matrix corresponding to the kth training video frame×H, H is a preset normalized Hadamard matrix, 2≤k≤K, and the prediction residual matrix corresponding to the kth training video frame is the difference between the current training video block of the kth training video frame and the current training video block of the (k-1)th training video frame.

[0233] S116, determining the maximum value of the standard deviation corresponding to each Hadamard matrix in the (K-1) Hadamard matrices as the frequency domain perception information of the current training video block, wherein the standard deviation corresponding to each Hadamard matrix is ​​the standard deviation of all pixels in each Hadamard matrix.

[0234] In some embodiments, a Hadamard matrix is ​​an n-order square matrix consisting of +1 and -1 elements and satisfying Hn*Hn'=nI (where Hn' is the transpose of Hn and I is a unit square matrix).

[0235] In some embodiments, the Hadamard matrix corresponding to the k-th training video frame is H×the prediction residual matrix corresponding to the k-th training video frame×H, which can be expressed by the following formula (15):

[0236] Hadamard b，k =H(B b，k -B b，k-1 )H (15)

[0237] Among them, Hadamard b，krepresents the Hadamard matrix corresponding to the current training video block b and the kth training video frame, H is the preset normalized Hadamard matrix, 2≤k≤K, K is the total number of video frames, (B b，k -B b，k-1 ) represents the difference between the current training video block of the kth training video frame and the current training video block of the (k-1)th training video frame.

[0238] The maximum value of the standard deviation corresponding to each Hadamard matrix in the (K-1) Hadamard matrices is determined as the frequency domain perception information of the current training video block, which can be expressed by the following formula (16):

[0239] FPT b =max k {σ(Hadamard b,k )},k∈{1,...,K-1} (16)

[0240] Among them, FPT b represents the frequency domain perception information of the current training video block b, max k {σ(Hadamard b,k )} represents the maximum value of the standard deviation corresponding to each Hadamard matrix in the Hadamard matrices of the video block b and the video frame k.

[0241] In an exemplary embodiment, the target video frame is input into a visual attention model to obtain a visual importance level of each video block, including:

[0242] S121, input the target video frame into the visual attention model to perform the following processing operations on the target video frame through the visual attention model to obtain the visual importance level of each video block.

[0243] S122, performing saliency detection on the target video frame to obtain a saliency value of each pixel in the target video frame;

[0244] S123, determining an average value of the significance values ​​of all pixels in each video block as a significance value corresponding to each video block;

[0245] S124, normalizing the saliency value corresponding to each video block to obtain a normalized saliency value corresponding to each video block;

[0246] S125 , determining a visual importance level that matches a corresponding significance threshold and a normalized significance value corresponding to each video block in each visual importance level in a set of visual importance levels as a visual importance level of each video block.

[0247] In some embodiments, saliency detection is performed on the target video frame to obtain the saliency value of each pixel in the target video frame, and the average value of the saliency values ​​of all pixels in each video block is determined as the saliency value corresponding to each video block, which can be achieved by the following formula (17):

[0248]

[0249] Among them, S b represents the average value of the saliency value of the video block b, s(x,y) represents the saliency value of the current pixel, x represents the horizontal coordinate of the pixel in the video block, y represents the vertical coordinate of the pixel in the video block, and r is the size of the video block, which is set to 64 here.

[0250] In some embodiments, the significance value corresponding to each video block is normalized to obtain a normalized significance value corresponding to each video block, and the visual significance level that matches the significance threshold corresponding to each visual significance level in a set of visual significance levels and the normalized significance value corresponding to each video block is determined as the visual significance level of each video block, which can be implemented by the following formula (18):

[0251]

[0252] Among them, VI b represents the visual importance level of video block b, S b represents the average value of the saliency value of video block b, S max With S min Represents the maximum and minimum significance of video block b in a video frame; θ is used to represent VI b The threshold value of , 1, i+1, and I all represent the visual importance level (which can also be the JND level), and i and I are both positive integers greater than 2.

[0253] Here, when the visual importance level of a video block is 1, it indicates that the video block contains a background area. When the visual importance level of a video block is I (maximum value), it indicates that the video block contains the most prominent object in the video frame. The visual importance level is I, which can be set according to spatial perception information, frequency domain perception information, or according to video image content and video image texture, and this application does not limit this.

[0254] In some embodiments, in addition to the JND level of the video block, the visual importance of each video block can also be used as a basis for determination in bit allocation, wherein the above visual attention model can be constructed by following steps 1 to 3:

[0255] In step 1, a video background template is constructed.

[0256] In some examples, the input video frame (image) is segmented using a simple linear iterative clustering (SLIC) algorithm to generate super pixels. Then, a background sample set is constructed by extracting features from each boundary segment. Finally, two reconstruction errors are calculated for each region using dense and sparse representations, respectively.

[0257] Here, a simple linear iterative clustering algorithm can convert a color image into a 5-dimensional feature vector in CIELAB color space and XY coordinates, and then construct a distance metric for the 5-dimensional feature vector to locally cluster the image pixels; a simple linear iterative clustering algorithm can generate compact, approximately uniform superpixels, and has good computing speed, object contour preservation, and superpixel shape. A superpixel refers to an irregular pixel block with certain visual significance composed of adjacent pixels with similar texture, color, brightness and other features; superpixels group pixels by using the similarity of features between pixels, and use a small number of superpixels instead of a large number of pixels to express image features (capture redundant information of the image), which greatly reduces the complexity of image post-processing, so it is used as a preprocessing step.

[0258] In step 2, the saliency of the video error is reconstructed.

[0259] In some examples, all image regions are reconstructed based on a background template. The reconstruction errors generated by dense and sparse appearance models are then smoothed using a propagation mechanism. For example, image segments are clustered into multiple clusters, and the segments in each cluster are sorted in descending order based on the reconstruction error; the segments in each cluster are processed sequentially by propagating the reconstruction error in each cluster; finally, pixel-level saliency is calculated using multi-scale reconstruction errors, followed by an object partial Gaussian refinement process.

[0260] Here, the object partial Gaussian refinement process is a Gaussian process. Gaussian process is a kind of random process in probability theory and mathematical statistics. It is an extension of multivariate Gaussian distribution and is applied in machine learning, signal processing and other fields.

[0261] In step 3, the visual importance level of each video block is calculated based on a Bayesian basis.

[0262] In some examples, a Bayesian integration method is used to combine the saliency maps constructed by dense reconstruction and sparse reconstruction to output a saliency value for each pixel, and then the average saliency value of each pixel block is obtained by the saliency value of each pixel to determine the visual importance level for each video block.

[0263] The calculation process of step 3 can refer to the above formula (17) and formula (18), which will not be repeated here.

[0264] refer to Figure 6 , Figure 6 It is an optional quantization parameter encoding schematic diagram provided in an embodiment of the present application; it includes stage one, stage two, stage three and stage four.

[0265] In stage one, a training data set is first obtained, where the training data set is the original video data. According to the original video data in the training data set, mapping is performed based on the spatial domain (spatial perception information), structural similarity, and frequency domain (frequency domain perception information) of the original video data to obtain a JND mapping diagram, wherein the JND mapping diagram includes a correspondence between the JND (level) and the quantization parameter (QP) of each video block of the video frame of the original video data (the value range of the quantization parameter is a positive integer of 1-51).

[0266] In the second stage, each video block in the video frame of the original video data and the JND of each video block are input into the initial CNN-based JND prediction training model to train the JND prediction training model and obtain a trained CNN-based JND prediction model.

[0267] In stage three, the test video sequence is input into the CNN-based JND prediction model and visual attention model respectively to obtain the predicted JND level of each video block and the visual importance level of each video block.

[0268] In stage 4, the target quantization parameter value of each video block is input, and the encoding quantization parameter value of each video block is obtained based on the quantization control algorithm according to the predicted JND level of each video block and the visual importance level of each video block.

[0269] In the related art, the traditional video coding method mainly improves the coding efficiency by removing spatial and temporal redundancy, but often ignores the visual perception of the human eye. In order to further effectively improve the coding efficiency, the embodiment of the present application provides a block-level perceptual video coding framework based on JND. First, the differences between different regions of each video frame are considered, and a JND level is assigned to each video block according to the SSIM, spatial and frequency domain characteristics of each block. Then, for each video block, the JND level and visual importance are estimated according to two different perceptual models. Finally, the quantization parameter value is reallocated to each block in combination with the JND level and visual importance; through the embodiment of the present application, the perceptual metrics and features can be effectively used to map the JND level from the frame to the block, and a block-level JND data set is provided to obtain a more robust perceptual modeling, which can greatly reduce the bit rate of perceptual video coding without losing visual quality.

[0270] According to another aspect of the embodiment of the present application, a parameter processing device for video coding is also provided, and the device is used to implement the parameter processing method for video coding provided in the above embodiment, which has been described and will not be repeated. As used below, the terms "module" and "unit" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0271] refer to Figure 7 , Figure 7 is a structural block diagram of an image processing device based on virtual reality according to an embodiment of the present application, such as Figure 7 As shown, the device comprises:

[0272] The first input unit 701 is used to input the target video frame to be encoded into the trained just perceptible distortion (JND) prediction model to obtain the predicted JND level of each video block in the target video frame, wherein the JND prediction model is used to predict the JND level of the video block in the input video frame;

[0273] A second input unit 702 is used to input the target video frame into the visual attention model to obtain a visual importance level of each video block, wherein the visual importance level of each video block is used to represent the degree of visual importance of each video block in the target video frame, and the visual attention model is used to predict the visual importance level of the video block in the input video frame based on the significance of the pixel points in the video block in the input video frame;

[0274] The adjusting unit 703 is configured to adjust the parameter value of the quantization parameter of each video block according to the predicted JND level of each video block and the visual importance level of each video block to obtain a target parameter value corresponding to each video block.

[0275] It should be noted that the first input unit 701 in this embodiment can be used to execute the above step S301, the second input unit 702 in this embodiment can be used to execute the above step S302, and the adjustment unit 703 in this embodiment can be used to execute the above step S303.

[0276] Through the embodiment of the present application, according to the trained just perceptible distortion JND prediction model, the JND level of the video block is predicted based on the visual parameters of the video block in the input video frame, and the predicted JND level of each video block in the target video frame is obtained, thereby, the predicted JND level is assigned to each video block to obtain the block-level JND data set, and the accuracy of visual redundancy estimation is improved; the target video frame is input into the visual attention model to obtain the visual importance level of each video block, thereby, each video block with different visual importance can be accurately identified from the target video frame; and then the parameter value of the quantization parameter of each video block is adjusted according to the predicted JND level of each video block and the visual importance level of each video block, thereby, the parameter value of the different quantization parameters can be accurately assigned to each video block in the video frame according to the perceptible quality level (JND level) and the visual importance, so as to maximize the elimination of imperceptible visual redundancy, and reduce the bit rate while maintaining the visual quality. Therefore, the parameter processing method of video encoding in the related art can solve the problem of low accuracy of visual redundancy estimation, and achieve the effect of maximizing the elimination of imperceptible visual redundancy while improving the accuracy of visual redundancy estimation.

[0277] In an exemplary embodiment, the adjusting unit comprises:

[0278] The first adjustment module is used to perform the following parameter value adjustment operation on each video block as the current video block to obtain a target parameter value corresponding to each video block, wherein the predicted JND level of the current video block is the i-th JND level among I JND levels, and the visual importance level of the current video block is the m-th visual importance level among M visual importance levels, wherein 1≤i≤I, 1≤m≤M:

[0279] A first acquisition module, configured to acquire an initial quantization parameter value corresponding to a current video block, wherein the initial quantization parameter value is an initial value of a quantization parameter configured for the current video block;

[0280] A first determining module is configured to, when the initial quantization parameter value is greater than the quantization parameter value corresponding to the current video block and the i-th JND level and is less than or equal to the quantization parameter value corresponding to the current video block and the (i+1)-th JND level, if (i+1+Mm)≤I, determine the quantization parameter value corresponding to the current video block and the (i+1+Mm)-th JND level as the target parameter value corresponding to the current video block;

[0281] The second determination module is used for comparing the quantization parameter value corresponding to the current video block and the I-th JND level with the initial quantization parameter value when the initial quantization parameter value is greater than the quantization parameter value corresponding to the current video block and the i-th JND level and is less than or equal to the quantization parameter value corresponding to the current video block and the (i+1)-th JND level, if I<(i+1+Mm), to determine the maximum value therebetween as the target parameter value corresponding to the current video block.

[0282] In an exemplary embodiment, the above device further comprises:

[0283] The first determination unit is used to, after obtaining an initial quantization parameter value corresponding to the current video block, determine the quantization parameter value corresponding to the current video block and the first JND level as a target parameter value corresponding to the current video block when the initial quantization parameter value is less than or equal to the quantization parameter value corresponding to the current video block and the first JND level.

[0284] In an exemplary embodiment, the adjusting unit comprises:

[0285] A second acquisition module is used to acquire a quantization parameter value corresponding to each video block, wherein the quantization parameter value corresponding to each video block is an initial value of the quantization parameter configured for each video block;

[0286] A third determination module determines an offset value of a quantization parameter corresponding to each video block according to a visual importance level of each video block, wherein the offset value of the quantization parameter corresponding to a video block is negatively correlated with the visual importance level of the video block;

[0287] The fourth determination module determines the sum of the quantization parameter value corresponding to each video block and the offset value of the quantization parameter corresponding to each video block as the target parameter value corresponding to each video block.

[0288] In an exemplary embodiment, the first input unit includes:

[0289] The first input module is used to input the target video frame to be encoded and each video block in the target video frame into the trained JND prediction model respectively, and obtain the predicted JND level of each video block output by the JND prediction model.

[0290] In an exemplary embodiment, the first input unit 701 includes:

[0291] A second input module is used to input the target video frame and each video block into a set of convolutional layers of the trained JND prediction model, and fuse the features extracted by the last convolutional layer in the set of convolutional layers into spatial features corresponding to each video block through the cascade layer of the JND prediction model;

[0292] The third input module is used to input each video block into the frequency domain feature extraction layer of the JND prediction model, and fuse the frequency domain perception features extracted by the frequency domain feature extraction layer into energy features corresponding to each video block through a cascade layer;

[0293] The fourth input module is used to input the spatial features corresponding to each video block and the energy features corresponding to each video block into the first fully connected layer, the long short-term memory network and the second fully connected layer connected in series in the JND prediction model to obtain the predicted JND level of each video block output by the JND prediction model.

[0294] In an exemplary embodiment, the second input module comprises:

[0295] The input subunit is used to input the target video frame into the first convolution layer in a group of convolution layers, and input the output of the first convolution layer together with each video block into the second convolution layer in a group of convolution layers after passing through an average pooling layer, so as to obtain the features extracted by the last convolution layer, wherein the convolution layers in a group of convolution layers are connected in series in sequence, and the outputs of the second convolution layer and each subsequent convolution layer need to pass through a maximum pooling layer and then be input into the connected convolution layers, and the output of the last convolution layer passes through a maximum pooling layer and then is input into the cascade layer.

[0296] In an exemplary embodiment, the above device further comprises:

[0297] A first acquisition unit is used to acquire a training video sequence and annotation information corresponding to the training video sequence before inputting a target video frame to be encoded into a trained just perceptible distortion (JND) prediction model, wherein the training video sequence includes K training video frames, each of the K training video frames is divided into L training video blocks in the same division method, and the annotation information corresponding to the training video sequence includes a JND level identifier of each training video block of each training video frame, and the JND level identifier of each training video block of each training video frame is used to identify the JND level of each training video block of each training video frame, and K and L are both positive integers greater than or equal to 2;

[0298] The using unit is used to perform multiple rounds of model training on the initial JND prediction model using the training video sequence and the annotation information corresponding to the training video sequence to obtain a trained JND prediction model.

[0299] In an exemplary embodiment, the using unit includes:

[0300] The first adjustment module is used to perform the following parameter adjustment operation on each training video block of each training video frame as the current training video block of the current training video frame in turn during a round of model training, wherein the JND level identified by the JND level identifier of the current training video block is the first JND level, and the loss function of the JND prediction model is a preset loss function:

[0301] a second input module, configured to input a current training video frame and a current training video block into a current JND prediction model, and obtain a level prediction result output by the current JND prediction model, wherein the level prediction result is used to indicate a second JND level of the current training video block predicted by the current JND prediction model;

[0302] A fifth determining module, configured to determine a function value corresponding to a preset loss function and a first JND level and a second JND level, and obtain a first function value;

[0303] The second adjustment module is used to adjust the model parameters of the current JND prediction model based on the first function value to obtain an adjusted current JND prediction model, wherein the JND level of the current training video block predicted by the adjusted current JND prediction model is a third JND level, and the first function value is greater than the function value of the preset loss function corresponding to the first JND level and the third JND level.

[0304] In an exemplary embodiment, the above device further comprises:

[0305] A second acquisition unit is used to acquire spatial perception information of each training video block of the training video sequence before acquiring the training video sequence and the annotation information corresponding to the training video sequence, wherein the spatial perception information of each training video block is used to represent the texture complexity of each training video block;

[0306] A third acquisition unit is used to acquire frequency domain perception information of each training video block of the training video sequence, wherein the frequency domain perception information of each training video block is used to represent the frequency domain energy of each training video block;

[0307] a second determining unit, configured to determine a perceptual parameter value of each training video block according to the spatial perceptual information of each training video block and the frequency domain perceptual information of each training video block, wherein the perceptual parameter value of each training video block is used to represent the tolerance of each training video block to distortion;

[0308] The third determination unit is configured to perform the following parameter value determination operation on each training video block as a current training video block to obtain a quantization parameter value corresponding to each training video block and each JND level in a set of JND levels:

[0309] a fourth determining unit, configured to determine a distortion value corresponding to the current training video block and each quantization parameter value within the quantization parameter value range, wherein the distortion value corresponding to the current training video block and each quantization parameter value is a distortion of an encoded video block obtained by encoding the current training video block using each quantization parameter value, which is determined based on structural similarity, relative to the current training video block;

[0310] a fifth determining unit, configured to determine a perceptual degradation metric value corresponding to the current training video block and each quantization parameter value by multiplying a distortion value corresponding to the current training video block and each quantization parameter value by a current perceptual coefficient, wherein the current perceptual coefficient is a product of the perceptual parameter value of the current training video block and the first coefficient plus a second coefficient;

[0311] a sixth determining unit, configured to determine a maximum distortion value corresponding to the training video sequence and each JND level, wherein the maximum distortion value corresponding to each JND level is a maximum value among distortion values ​​corresponding to each training video block of the training video sequence and a preset quantization parameter value of each JND level;

[0312] a seventh determination unit, configured to determine, based on the perceptual degradation metric value corresponding to each quantization parameter value of the current training video block and the maximum distortion value corresponding to each JND level, the quantization parameter value corresponding to the current training video block and each JND level being the maximum value among the quantization parameter values ​​whose corresponding perceptual degradation metric value is less than or equal to the maximum distortion value corresponding to each JND level.

[0313] In an exemplary embodiment, the second determining unit includes:

[0314] A first normalization module is used to normalize the spatial perception information of each training video block to obtain the normalized spatial complexity of each training video block;

[0315] A second normalization module is used to normalize the frequency domain perception information of each training video block to obtain a normalized frequency domain energy value of each training video block;

[0316] The sixth determination module determines the ratio of the normalized frequency domain energy value of each training video block to the normalized spatial complexity of each training video block as the perceptual parameter value of each training video block.

[0317] In an exemplary embodiment, the second acquiring unit includes:

[0318] The first execution module is used to perform the following acquisition operations on each training video block of the training video sequence as a current training video block to obtain spatial perception information of each training video block: respectively apply Scharr filters to the current training video blocks of K training video frames for filtering processing to obtain K Scharr filter blocks; determine the maximum value of the standard deviation corresponding to each Scharr filter block in the K Scharr filter blocks as the spatial perception information of the current training video block, wherein the standard deviation corresponding to each Scharr filter block is the standard deviation of all pixels in each Scharr filter block;

[0319] In an exemplary embodiment, the third obtaining unit includes:

[0320] The second execution module is used to perform the following acquisition operation on each training video block of the training video sequence as the current training video block to obtain the frequency domain perception information of each training video block: respectively calculate the Hadamard matrices corresponding to each training video frame from the second training video frame to the Kth training video frame in the K training video frames to obtain (K-1) Hadamard matrices, wherein the Hadamard matrix corresponding to the kth training video frame is H×the prediction residual square matrix corresponding to the kth training video frame×H, H is a preset normalized Hadamard matrix, 2≤k≤K, and the prediction residual square matrix corresponding to the kth training video frame is the difference between the current training video block of the kth training video frame and the current training video block of the (k-1)th training video frame; determine the maximum value of the standard deviation corresponding to each Hadamard matrix in the (K-1) Hadamard matrices as the frequency domain perception information of the current training video block, wherein the standard deviation corresponding to each Hadamard matrix is ​​the standard deviation of all pixels in each Hadamard matrix.

[0321] In an exemplary embodiment, the second input unit includes:

[0322] The third input module is used to input the target video frame into the visual attention model, so as to perform the following processing operations on the target video frame through the visual attention model to obtain the visual importance level of each video block:

[0323] A detection module is used to perform saliency detection on the target video frame to obtain a saliency value of each pixel in the target video frame;

[0324] a seventh determination module, configured to determine a significance value corresponding to each video block according to an average of the significance values ​​of all pixels in each video block;

[0325] The eighth determination module is used to determine the visual importance level of each video block by matching the corresponding significance threshold in each visual importance level in a set of visual importance levels with the significance value corresponding to each video block.

[0326] Optional examples of this embodiment can refer to the examples shown in the above-mentioned virtual reality-based image processing method, which will not be described in detail in this embodiment.

[0327] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0328] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0329] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0330] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit 801, various functions provided by the embodiments of the present application are executed. The above serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0331] Figure 8 The computer system structure block diagram of the electronic device used to implement the embodiment of the present application is schematically shown. Figure 8As shown, the computer system 800 includes a central processing unit 801 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 802 (ROM) or the program loaded from the storage part 808 to the random access memory 803 (RAM). Various programs and data required for system operation are also stored in the random access memory 803. The central processing unit 801, the read-only memory 802 and the random access memory 803 are connected to each other through a bus 804. The input / output interface 805 (Input / Output interface, i.e., I / O interface) is also connected to the bus 804.

[0332] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a local area network card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0333] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processor 801, various functions defined in the system of the present application are executed.

[0334] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0335] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0336] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the input / output resource pool, and the input / output device is connected to the input / output resource pool.

[0337] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0338] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0339] The above description is only the preferred embodiment of the present application and is not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A parameter processing method for video coding, characterized in that: include: Inputting a target video frame to be encoded into a trained just perceptible distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame, wherein the JND prediction model is used to predict the JND level of the video block in the input video frame; Inputting the target video frame into a visual attention model to obtain a visual importance level of each video block, wherein the visual importance level of each video block is used to represent the degree of visual importance of each video block in the target video frame, and the visual attention model is used to predict the visual importance level of the video block in the input video frame based on the significance of the pixel points in the video block in the input video frame; According to the predicted JND level of each video block and the visual importance level of each video block, a parameter value of a quantization parameter of each video block is adjusted to obtain a target parameter value corresponding to each video block.

2. The method according to claim 1, characterized in that The step of adjusting the parameter value of the quantization parameter of each video block according to the predicted JND level of each video block and the visual importance level of each video block to obtain a target parameter value corresponding to each video block includes: The following parameter value adjustment operation is performed on each of the video blocks as a current video block to obtain a target parameter value corresponding to each of the video blocks, wherein the predicted JND level of the current video block is the i-th JND level among I JND levels, and the visual importance level of the current video block is the m-th visual importance level among M visual importance levels, wherein 1≤i≤I, 1≤m≤M: Acquire an initial quantization parameter value corresponding to the current video block, wherein the initial quantization parameter value is an initial value of a quantization parameter configured for the current video block; When the initial quantization parameter value is greater than the quantization parameter value corresponding to the current video block and the i-th JND level and less than or equal to the quantization parameter value corresponding to the current video block and the (i+1)-th JND level, if (i+1+Mm)≤I, the quantization parameter value corresponding to the current video block and the (i+1+Mm)-th JND level is determined as the target parameter value corresponding to the current video block; When the initial quantization parameter value is greater than the quantization parameter value corresponding to the current video block and the i-th JND level and is less than or equal to the quantization parameter value corresponding to the current video block and the (i+1)-th JND level, if I<(i+1+Mm), the quantization parameter value corresponding to the current video block and the I-th JND level is compared with the initial quantization parameter value, and the maximum value therebetween is determined as the target parameter value corresponding to the current video block.

3. The method according to claim 2, characterized in that After obtaining the initial quantization parameter value corresponding to the current video block, the method further includes: When the initial quantization parameter value is less than or equal to the quantization parameter value corresponding to the current video block and the first JND level, the quantization parameter value corresponding to the current video block and the first JND level is determined as a target parameter value corresponding to the current video block.

4. The method according to claim 1, characterized in that: The step of adjusting the parameter value of the quantization parameter of each video block according to the predicted JND level of each video block and the visual importance level of each video block to obtain a target parameter value corresponding to each video block includes: Acquire a quantization parameter value corresponding to each video block, wherein the quantization parameter value corresponding to each video block is an initial value of the quantization parameter configured for each video block; Determining an offset value of a quantization parameter corresponding to each video block according to the visual importance level of each video block, wherein the offset value of the quantization parameter corresponding to a video block is negatively correlated with the visual importance level of the video block; The sum of the quantization parameter value corresponding to each video block and the offset value of the quantization parameter corresponding to each video block is determined as the target parameter value corresponding to each video block.

5. The method according to claim 1, characterized in that The step of inputting the target video frame to be encoded into a trained just perceptible distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame comprises: The target video frame to be encoded and each video block in the target video frame are respectively input into the trained JND prediction model to obtain a predicted JND level of each video block output by the JND prediction model.

6. The method according to claim 1, characterized in that The step of inputting the target video frame to be encoded into a trained just perceptible distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame comprises: Inputting the target video frame and each video block into a set of convolutional layers of the trained JND prediction model, and fusing the features extracted by the last convolutional layer in the set of convolutional layers into spatial features corresponding to each video block through the cascade layer of the JND prediction model; Inputting each video block into the frequency domain feature extraction layer of the JND prediction model, and fusing the frequency domain perceptual features extracted by the frequency domain feature extraction layer into energy features corresponding to each video block through the cascade layer; The spatial features corresponding to each video block and the energy features corresponding to each video block are input into the first fully connected layer, the long short-term memory network, and the second fully connected layer connected in series in the JND prediction model to obtain the predicted JND level of each video block output by the JND prediction model.

7. The method according to claim 6, characterized in that The inputting the target video frame and each video block into a set of convolutional layers of the trained JND prediction model comprises: The target video frame is input into the first convolution layer in the group of convolution layers, and the output of the first convolution layer is input into the second convolution layer in the group of convolution layers together with each video block after passing through an average pooling layer to obtain the features extracted by the last convolution layer, wherein the convolution layers in the group of convolution layers are connected in series in sequence, the outputs of the second convolution layer and each subsequent convolution layer need to pass through a maximum pooling layer and then be input into the connected convolution layers, and the output of the last convolution layer is input into the cascade layer after passing through a maximum pooling layer.

8. The method according to claim 1, characterized in that Before inputting the target video frame to be encoded into the trained just perceptible distortion (JND) prediction model, the method further includes: Acquire a training video sequence and annotation information corresponding to the training video sequence, wherein the training video sequence includes K training video frames, each of the K training video frames is divided into L training video blocks in the same division method, and the annotation information corresponding to the training video sequence includes a JND level identifier of each training video block of each training video frame, and the JND level identifier of each training video block of each training video frame is used to identify the JND level of each training video block of each training video frame, and K and L are both positive integers greater than or equal to 2; The initial JND prediction model is trained multiple times using the training video sequence and the annotation information corresponding to the training video sequence to obtain the trained JND prediction model.

9. The method according to claim 8, characterized in that The method of performing multiple rounds of model training on the initial JND prediction model using the training video sequence and the annotation information corresponding to the training video sequence to obtain the trained JND prediction model includes: In the process of performing a round of model training, each training video block of each training video frame is sequentially used as a current training video block of a current training video frame to perform the following parameter adjustment operation, wherein the JND level identified by the JND level identifier of the current training video block is a first JND level, and the loss function of the JND prediction model is a preset loss function: Inputting the current training video frame and the current training video block into a current JND prediction model to obtain a level prediction result output by the current JND prediction model, wherein the level prediction result is used to indicate a second JND level of the current training video block predicted by the current JND prediction model; Determine a function value of the preset loss function corresponding to the first JND level and the second JND level to obtain a first function value; Based on the first function value, the model parameters of the current JND prediction model are adjusted to obtain the adjusted current JND prediction model, wherein the JND level of the current training video block predicted by the adjusted current JND prediction model is the third JND level, and the first function value is greater than the function value of the preset loss function corresponding to the first JND level and the third JND level.

10. The method according to claim 8, characterized in that Before acquiring the training video sequence and the annotation information corresponding to the training video sequence, the method further includes: Acquire spatial perception information of each training video block of the training video sequence, wherein the spatial perception information of each training video block is used to represent the texture complexity of each training video block; Acquire frequency domain perception information of each training video block of the training video sequence, wherein the frequency domain perception information of each training video block is used to represent frequency domain energy of each training video block; Determining a perceptual parameter value of each training video block according to the spatial perceptual information of each training video block and the frequency domain perceptual information of each training video block, wherein the perceptual parameter value of each training video block is used to represent the tolerance of each training video block to distortion; The following parameter value determination operation is performed on each training video block as a current training video block to obtain a quantization parameter value corresponding to each training video block and each JND level in a set of JND levels: Determine a distortion value corresponding to the current training video block and each quantization parameter value within a quantization parameter value range, wherein the distortion value corresponding to the current training video block and each quantization parameter value is a distortion of an encoded video block obtained by encoding the current training video block using each quantization parameter value, determined based on structural similarity, relative to the current training video block; Determine a perceptual degradation metric value corresponding to the current training video block and each quantization parameter value by multiplying a distortion value corresponding to the current training video block and each quantization parameter value by a current perceptual coefficient, wherein the current perceptual coefficient is a product of the perceptual parameter value of the current training video block and a first coefficient plus a second coefficient; Determining a maximum distortion value corresponding to the training video sequence and each JND level, wherein the maximum distortion value corresponding to each JND level is a maximum value among distortion values ​​corresponding to each training video block of the training video sequence and a preset quantization parameter value of each JND level; According to the perceptual degradation metric value corresponding to the current training video block and each quantization parameter value and the maximum distortion value corresponding to each JND level, the quantization parameter value corresponding to the current training video block and each JND level is determined, wherein the quantization parameter value corresponding to the current training video block and each JND level is the maximum value of the quantization parameter values ​​whose corresponding perceptual degradation metric value is less than or equal to the maximum distortion value corresponding to each JND level.

11. The method according to claim 10, characterized in that The determining, according to the spatial perception information of each training video block and the frequency domain perception information of each training video block, the perception parameter value of each training video block comprises: Normalizing the spatial perception information of each training video block to obtain a normalized spatial complexity of each training video block; Normalizing the frequency domain perception information of each training video block to obtain a normalized frequency domain energy value of each training video block; The ratio of the normalized frequency domain energy value of each training video block to the normalized spatial complexity of each training video block is determined as the perceptual parameter value of each training video block.

12. The method according to claim 10, characterized in that The acquiring of the spatial perception information of each training video block of the training video sequence comprises: taking each training video block of the training video sequence as a current training video block and performing the following acquisition operations to acquire the spatial perception information of each training video block: applying a Scharr filter to the current training video blocks of the K training video frames for filtering processing to acquire K Scharr filter blocks; determining the maximum value of the standard deviation corresponding to each Scharr filter block in the K Scharr filter blocks as the spatial perception information of the current training video block, wherein the standard deviation corresponding to each Scharr filter block is the standard deviation of all pixels in each Scharr filter block; The acquiring of the frequency domain perception information of each training video block of the training video sequence includes: taking each training video block of the training video sequence as a current training video block and performing the following acquisition operation to acquire the frequency domain perception information of each training video block: respectively calculating the Hadamard matrix corresponding to each training video frame from the second training video frame to the Kth training video frame in the K training video frames to acquire (K-1) Hadamard matrices, wherein the Hadamard matrix corresponding to the kth training video frame is H×the predicted matrix corresponding to the kth training video frame A prediction residual matrix × H, H is a preset normalized Hadamard matrix, 2≤k≤K, and the prediction residual matrix corresponding to the kth training video frame is the difference between the current training video block of the kth training video frame and the current training video block of the (k-1)th training video frame; the maximum value of the standard deviation corresponding to each Hadamard matrix in the (K-1) Hadamard matrices is determined as the frequency domain perception information of the current training video block, wherein the standard deviation corresponding to each Hadamard matrix is ​​the standard deviation of all pixels in each Hadamard matrix.

13. The method according to any one of claims 1 to 12, characterized in that Inputting the target video frame into a visual attention model to obtain a visual importance level of each video block includes: The target video frame is input into the visual attention model, so as to perform the following processing operations on the target video frame through the visual attention model to obtain the visual importance level of each video block: Performing saliency detection on the target video frame to obtain a saliency value of each pixel in the target video frame; Determining a significance value corresponding to each video block according to an average value of significance values ​​of all pixels in each video block; In each visual importance level in a set of visual importance levels, a visual importance level that matches a significance threshold and a significance value corresponding to each video block is determined as a visual importance level of each video block.

14. A parameter processing device for video coding, characterized in that: include: A first input unit is used to input a target video frame to be encoded into a trained just noticeable distortion (JND) prediction model to obtain a predicted JND level of each video block in the target video frame, wherein the JND prediction model is used to predict the JND level of the video block in the input video frame; A second input unit is used to input the target video frame into a visual attention model to obtain a visual importance level of each video block, wherein the visual importance level of each video block is used to represent the degree of visual importance of each video block in the target video frame, and the visual attention model is used to predict the visual importance level of the video block in the input video frame based on the significance of the pixel points in the video block in the input video frame; The adjusting unit is used to adjust the parameter value of the quantization parameter of each video block according to the predicted JND level of each video block and the visual importance level of each video block to obtain a target parameter value corresponding to each video block.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 13 when executed by a processor.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 13 are implemented.

Images