Video frame adjusting method and electronic device

Abstract

The application discloses a video frame adjusting method and an electronic device. The method establishes time domain correlation through a dense motion field between saliency maps, thereby generating a predicted saliency map. This process simulates the continuity of visual attention in time sequence, and provides a more accurate benchmark for evaluating the difference between the actual saliency map and the predicted saliency map. The actual saliency map and the predicted saliency map of a frame image are analyzed to determine the offset of the quantization parameter of the frame image, and the deviation between the actual distribution of the current frame image attention and the prediction is quantified. Based on this, the code rate is adjusted, and the visual experience is optimized.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video frame adjustment method and electronic device. Background Technology

[0002] In video compression coding, the allocation of video frame bitrate mainly depends on spatial complexity and frame type, without considering the issue of temporal attention distribution. This leads to distortion and other problems when there are camera transitions or new objects entering the video, thus affecting video quality. Summary of the Invention

[0003] In view of this, this application provides a video frame adjustment method and an electronic device, the specific solution of which is as follows: A video frame adjustment method, comprising: Obtain a first saliency map corresponding to the frame image at the first moment in the video and a second saliency map corresponding to the frame image at the second moment, wherein the second moment is the moment before the first moment and adjacent to the first moment; Determine the dense motion field between the first saliency map and the second saliency map; Based on the dense motion field and the second saliency map, the predicted saliency map corresponding to the frame image at the first moment is determined; The offset of the quantization parameters of the frame image at the first time moment is determined based on the first saliency map at the first time moment and the predicted saliency map. The bitrate of the frame image at the first moment is adjusted based on the offset of the quantization parameters of the frame image at the first moment.

[0004] Furthermore, determining the predicted saliency map corresponding to the frame image at the first moment based on the dense motion field and the second saliency map includes: According to the dense motion field, the second saliency map is interpolated to obtain a motion-compensated saliency map, and the motion-compensated saliency map is determined as the predicted saliency map corresponding to the frame image at the first time.

[0005] Furthermore, the step of interpolating the second saliency map according to the dense motion field to obtain a motion-compensated saliency map includes: Determine the initial coordinates corresponding to each pixel position in the second saliency map; Based on the motion displacement of each pixel position in the dense motion field and the initial coordinates of the corresponding pixel, the offset coordinates of each pixel position are determined. The motion-compensated saliency map is determined based on the offset coordinates of each pixel position.

[0006] Furthermore, the step of interpolating the second saliency map according to the dense motion field to obtain a motion-compensated saliency map includes: Determine the target coordinates of each pixel position in the motion-compensated saliency map to be generated, wherein the motion-compensated saliency map to be generated is a saliency map after motion compensation of the second saliency map; Based on each target coordinate and the displacement corresponding to the target coordinate in the dense motion field, determine the source coordinates corresponding to each target coordinate in the second saliency map; If the source coordinates are non-integer coordinates, determine the number of integer coordinates of targets adjacent to the source coordinates in the second saliency map; Based on the pixel value at each integer coordinate in the integer coordinates of the target quantity, determine the pixel value at each target coordinate in the motion-compensated saliency map to be generated.

[0007] Further, determining the offset of the quantization parameters of the frame image at the first time step based on the first saliency map at the first time step and the predicted saliency map includes: The target temporal weights of the frame images at the first time point are determined based on the first saliency map at the first time point and the predicted saliency map. By utilizing the correspondence between the target temporal weight and the offset, the offset of the quantization parameters of the frame image at the first moment corresponding to the target temporal weight is determined.

[0008] Furthermore, determining the target temporal weight of the frame image at the first time step based on the first saliency map at the first time step and the predicted saliency map includes: A first value is determined based on the pixel value at the target location in the first saliency map and the pixel value at the target location in the predicted saliency map. The first value is used to characterize the difference between the first saliency map and the predicted saliency map. A second value is determined based on the first saliency map and the second saliency map. The second value is used to characterize the amount of change between the attention value of the first saliency map and the attention value of the second saliency map. The target temporal weight of the frame image at the first moment is determined based at least on the first value and the second value.

[0009] Furthermore, determining the first value based on the pixel value at the target location in the first saliency map and the pixel value at the target location in the predicted saliency map includes: The difference between each pixel position in the first saliency map and the pixel value at the corresponding pixel position in the predicted saliency map is determined to obtain the prediction residual for each pixel position; The absolute value of the prediction residual for each pixel position is averaged, and the resulting average value is determined as the first value.

[0010] Furthermore, determining the second value based on the first saliency map and the second saliency map includes: Determine the average value of all pixels in the first saliency map and the average value of all pixels in the second saliency map; The second value is determined based on the difference between the average value of all pixels in the first saliency map and the average value of all pixels in the second saliency map.

[0011] Furthermore, determining the offset of the quantization parameters of the frame image at the first moment corresponding to the target temporal weight by utilizing the correspondence between the target temporal weight and the offset includes: Determine the target temporal weight of each frame in the video to obtain the target temporal weight sequence of the video; By utilizing the correspondence between the target temporal weight and the offset, the offset of the quantization parameter of each frame image in the video corresponding to the target temporal weight is determined.

[0012] An electronic device, comprising: A processor is configured to obtain a first saliency map corresponding to a frame image at a first moment in a video and a second saliency map corresponding to a frame image at a second moment, wherein the second moment is a moment prior to and adjacent to the first moment; determine a dense motion field between the first saliency map and the second saliency map; determine a predicted saliency map corresponding to the frame image at the first moment based on the dense motion field and the second saliency map; determine an offset of the quantization parameters of the frame image at the first moment based on the first saliency map and the predicted saliency map; and adjust the bitrate of the frame image at the first moment based on the offset of the quantization parameters of the frame image at the first moment. The memory is used to store the programs required by the processor to perform the above-described processing.

[0013] As can be seen from the above technical solutions, the video frame adjustment method and electronic device disclosed in this application obtain saliency maps corresponding to two adjacent frames in a video, determine the saliency map corresponding to the previous frame as the second saliency map, and determine the saliency map corresponding to the next frame as the first saliency map. Based on the dense motion field between the first and second saliency maps and the second saliency map, the predicted saliency map of the next frame is determined. Based on the predicted saliency map and the first saliency map of the next frame, the offset of the quantization parameters of the next frame is determined, and the bitrate of the next frame is adjusted based on the offset. This solution establishes a temporal correlation through the dense motion field between saliency maps, thereby generating a predicted saliency map. This process simulates the temporal continuity of visual attention, providing a more accurate benchmark for evaluating the difference between the actual saliency map and the predicted saliency map. By analyzing the actual and predicted saliency maps of the frame image, the offset of the quantization parameters of the frame image is determined, quantifying the deviation between the actual distribution and prediction of attention in the current frame image, and the bitrate is adjusted based on this, thus optimizing the visual experience. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a video frame adjustment method disclosed in an embodiment of this application; Figure 2 A flowchart illustrating another video frame adjustment method disclosed in an embodiment of this application; Figure 3 This is a schematic diagram of determining pixel Q in a predicted saliency map based on a second saliency map, as disclosed in an embodiment of this application. Figure 4 This is a schematic diagram illustrating a method for determining the prediction saliency data of pixel Q in a prediction saliency map, as disclosed in an embodiment of this application. Figure 5 This is a flowchart illustrating yet another video frame adjustment method disclosed in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a video frame adjustment system disclosed in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0016] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0017] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0018] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0019] This application discloses a video frame adjustment method, the flowchart of which is shown below. Figure 1 As shown, it includes: Step S11: Obtain the first saliency map corresponding to the frame image at the first moment in the video and the second saliency map corresponding to the frame image at the second moment, where the second moment is the moment before the first moment and adjacent to the first moment; Step S12: Determine the dense motion field between the first saliency graph and the second saliency graph; Step S13: Determine the predicted saliency map corresponding to the frame image at the first moment based on the dense motion field and the second saliency map; Step S14: Determine the offset of the quantization parameters of the frame image at the first time based on the first saliency map and the predicted saliency map at the first time. Step S15: Adjust the bitrate of the frame image at the first moment based on the offset of the quantization parameters of the frame image at the first moment.

[0020] In video compression coding, the allocation of video frame bitrate mainly depends on spatial complexity and frame type, without considering the issue of temporal attention distribution. This leads to distortion and other problems when there are camera transitions or new objects entering the video, thus affecting video quality.

[0021] Based on this, this scheme obtains the saliency maps corresponding to two adjacent frames in the video, namely the second saliency map corresponding to the previous moment (second moment) and the first saliency map corresponding to the next moment (first moment). It then determines the dense motion field between the first and second saliency maps, and obtains the predicted saliency map for the first moment based on the dense motion field and the second saliency map. The first saliency map for the first moment is compared with the predicted saliency map to determine the offset of the quantization parameters of the frame image at the first moment, so that the bitrate of the frame image at the first moment can be adjusted based on the offset of the quantization parameters. This scheme does not require the source video to be involved when adjusting the bitrate of frame images in the video using saliency maps, reducing the complexity of bitrate adjustment. Furthermore, by establishing a temporal correlation through the dense motion field between saliency maps, a predicted saliency map is generated. This process simulates the temporal continuity of visual attention, providing a more accurate benchmark for evaluating the difference between the actual and predicted saliency maps, thus ensuring the accuracy of bitrate adjustment and improving the visual experience of the video.

[0022] After determining the video whose bitrate of image frames needs to be adjusted, the saliency map corresponding to each frame of the video can be determined so that analysis can be performed based on a certain frame and its previous frame, thereby adjusting the bitrate of that frame.

[0023] A saliency map is a grayscale image used to represent the degree to which different regions in an image or video frame attract the human eye. Its core function is to quantify the distribution of visual attention. A saliency map is usually a grayscale image of the same size as the original frame image. High grayscale values ​​in the saliency map indicate high saliency, which is the region that the human eye is most likely to pay attention to, such as moving objects and faces. Low grayscale values ​​in the saliency map indicate low saliency, which is the background region that the human eye is less likely to pay attention to.

[0024] The first saliency map corresponding to the first frame image in the video. And the second saliency map corresponding to the frame image at the second time step. Taking this as an example, the first moment can be the current moment or any moment in the video, and the second moment is the moment before the first moment and adjacent to the first moment. Then the frame image of the first moment and the frame image of the second moment are adjacent images in the video, and the frame image of the second moment is located before the frame image of the first moment in the time domain.

[0025] In obtaining the first saliency map and the second significance plot Then, the first saliency map can be determined. and the second significance plot For dense motion fields, the DIS (Dense Inverse Search) optical flow algorithm can be used to analyze the first saliency map. and the second significance plot Calculations were performed to obtain the dense sports field. Dense sports fields It contains each pixel in the first saliency map and the second significance plot Two-dimensional motion vectors between.

[0026] The process of obtaining a dense sports field can be specifically described as follows: For the first saliency graph and the second significance plot Five image pyramids were constructed (the bottom layer is the saliency map at the original resolution, and the width and height of the image are reduced to half of the original for each layer above, with the top layer having the lowest resolution). For each image layer from top to bottom, two steps are performed: sparse inverse search and dense variational refinement. Specifically, for the sparse inverse search, starting from the first saliency map... Randomly select N 8×8 or 16×16 square blocks (i.e., patches) from the current layer and place them in the second saliency graph. The best-matching patch is searched within the same layer to minimize the sum of squared differences between pixels within the patch. The relative displacement of the two patches within the current layer's image in two dimensions is then determined as the motion vector of the current patch's center point, thus yielding a sparse motion vector field. For dense variational refinement, bilinear interpolation is used to calculate the motion vectors of all pixels in the current layer except the patch center, forming a coarse-dense motion field for the current layer. Keeping the motion vector of the patch center point unchanged, exponential moving average and gradient descent are used to smooth the motion vectors of other pixels, resulting in a finer-grained dense motion field for the current layer. After calculating the dense motion field of the current layer, the nearest neighbor algorithm is used to upsample it to the image size of the next layer, which is then used as the initial motion vector for the sparse inverse search of the next layer. This ensures that when searching for patches in the next layer, the search can start from a specified position without having to traverse all pixels of the next layer from scratch, thus reducing the amount of computation. The above steps are repeated until the dense motion field of the original resolution layer saliency map is calculated.

[0027] After obtaining the dense motion field, the saliency map corresponding to the frame image at the first time can be predicted based on the dense motion field and the second saliency map at the second time to obtain the predicted saliency map. After obtaining the predicted saliency map corresponding to the frame image at the first time, the predicted saliency map and the first saliency map at the first time can be analyzed. Based on the difference between the predicted saliency map and the actual saliency map (the first saliency map), the offset of the quantization parameter is determined so that the bit rate of the frame image at the first time can be adjusted based on the offset.

[0028] Among these parameters, the quantization parameter QP is one of the most crucial parameters in video coding, directly determining the compression quality and bitrate of the video. QP is the process of mapping continuous transform coefficients (such as DCT coefficients, i.e., discrete cosine transform coefficients) with a large range of values ​​to finite discrete values ​​with a smaller range during video coding. QP is the step size control parameter for this quantization process. A small QP value indicates a small quantization step size, preserving more details, resulting in a higher bitrate, better video quality, and a larger file size. Conversely, a large QP value indicates a large quantization step size, preserving less detail, resulting in a lower bitrate, poorer video quality, and a smaller file size.

[0029] The video frame adjustment method disclosed in this embodiment can be implemented based on an encoder. After determining the offset of the quantization parameters of a certain frame image, the initial quantization parameter value determined by the encoder for that frame image is obtained. Then, the initial quantization parameter value is added to the obtained quantization parameter offset (the quantization parameter offset can be a positive value, in which case the bit rate is reduced; it can also be a negative value, in which case the bit rate is increased; or it can be 0, in which case the bit rate is not adjusted). The result is the final target quantization parameter value determined for that frame image. After the target quantization parameter value of the frame image is sent to the encoder, the encoder can automatically adjust the compression degree of the frame image based on the target quantization parameter value of the frame image, thereby accurately realizing the bit rate allocation.

[0030] The encoder can be a variety of different encoders, such as the SVT-AV1 encoder. Of course, this encoder is only an example and is not intended to limit its use.

[0031] The video frame adjustment method disclosed in this embodiment obtains saliency maps corresponding to two adjacent frames in a video. The saliency map corresponding to the previous frame is determined as the second saliency map, and the saliency map corresponding to the next frame is determined as the first saliency map. Based on the dense motion field between the first and second saliency maps and the second saliency map, a predicted saliency map for the next frame is determined. Based on the predicted saliency map and the first saliency map, the offset of the quantization parameters for the next frame is determined, and the bitrate of the next frame is adjusted based on this offset. This scheme establishes a temporal correlation through the dense motion field between saliency maps, thereby generating a predicted saliency map. This process simulates the temporal continuity of visual attention, providing a more accurate benchmark for evaluating the difference between the actual and predicted saliency maps. Analyzing the actual and predicted saliency maps of a frame image determines the offset of the quantization parameters for that frame image, quantifying the deviation between the actual and predicted distribution of attention in the current frame image, and adjusting the bitrate accordingly to optimize the visual experience.

[0032] This embodiment discloses a video frame adjustment method, the flowchart of which is as follows: Figure 2 As shown, it includes: Step S21: Obtain the first saliency map corresponding to the frame image at the first moment in the video and the second saliency map corresponding to the frame image at the second moment, where the second moment is the moment before the first moment and adjacent to the first moment; Step S22: Determine the dense motion field between the first saliency map and the second saliency map; Step S23: According to the dense motion field, the second saliency map is interpolated to obtain the saliency map after motion compensation of the second saliency map, and the motion-compensated saliency map is determined as the predicted saliency map corresponding to the frame image at the first time. Step S24: Determine the offset of the quantization parameters of the frame image at the first time based on the first saliency map and the predicted saliency map at the first time. Step S25: Adjust the bitrate of the frame image at the first moment based on the offset of the quantization parameters of the frame image at the first moment.

[0033] The process involves obtaining saliency maps for two adjacent frames in the video, specifically the first saliency map corresponding to the frame at the first time step and the second saliency map corresponding to the frame at the second time step (the second time step is earlier than the first time step). A dense motion field between the first and second saliency maps is determined, and a predicted saliency map corresponding to the frame at the first time step is calculated based on this dense motion field and the second saliency map. This allows for the determination of the quantization parameter offset for the frame at the first time step, based on both the first and predicted saliency maps, and the bitrate of the frame at the first time step is adjusted accordingly. Establishing a temporal correlation through the dense motion field between saliency maps generates the predicted saliency map. This process simulates the temporal continuity of visual attention, providing a more accurate benchmark for evaluating the difference between the actual and predicted saliency maps, ensuring the accuracy of bitrate adjustment, and optimizing the visual experience of the video.

[0034] Specifically, determining the predicted saliency map can be achieved by interpolating the second saliency map according to the dense motion field to obtain a motion-compensated saliency map, and then determining the motion-compensated saliency map as the predicted saliency map corresponding to the frame image at the first moment.

[0035] Obtain a dense motion field between the first saliency map and the second saliency map. This dense motion field contains pixel-level motion displacement information from the second saliency map to the first saliency map, that is, the displacement of each pixel position in the horizontal and vertical directions.

[0036] Subsequently, based on the dense motion field, the second saliency map is interpolated to obtain a motion-compensated saliency map, which is then used as the predicted saliency map corresponding to the frame image at the first time step. This predicted saliency map represents an estimate of the saliency map at the first time step based solely on the motion changes in the second saliency map. Specifically, the interpolation process involves determining the new positions of each pixel in the second saliency map after motion compensation based on the motion displacement information provided by the dense motion field, and assigning corresponding pixel values ​​at these new positions using interpolation.

[0037] Further interpolation processing can be specifically described as follows: Determine the initial coordinates corresponding to each pixel position in the second saliency map; determine the offset coordinates of each pixel position based on the motion displacement of each pixel position in the dense motion field and the corresponding initial coordinates of the pixel; determine the saliency map after motion compensation of the second saliency map based on the offset coordinates of each pixel position.

[0038] After obtaining the dense motion field between the first saliency map at the first time step and the second saliency map at the second time step, the motion compensation of the second saliency map is performed using the forward mapping method disclosed in this embodiment to generate the predicted saliency map at the first time step.

[0039] In the forward mapping method, the initial coordinates corresponding to each pixel position in the second saliency map are first determined. The initial coordinates are the position coordinates of each pixel in the second saliency map in its original image coordinate system, usually represented by the row and column index of the pixel position. For example, for a saliency map with a resolution of 1920×1080, the initial coordinates of the pixel position in the i-th row and j-th column are (i, j), where the value of i ranges from 0 to 1079, and the value of j ranges from 0 to 1919.

[0040] Based on the motion displacement of each pixel position in the dense motion field and the initial coordinates of the corresponding pixel, the offset coordinates of each pixel position are determined. Specifically, for the pixel located at position (i, j) in the second saliency map, this position corresponds to a two-dimensional motion displacement (u, v) in the dense motion field, where u represents the horizontal motion displacement of the pixel and v represents the vertical motion displacement of the pixel. The offset coordinates (i', j') of this pixel position are determined in the following way:

[0041] Each pixel position in the second saliency map has a corresponding offset coordinate. The pixel values ​​of each pixel position in the second saliency map are moved according to their offset coordinates to obtain the motion-compensated saliency map, which is the predicted saliency map at the first time step.

[0042] During the forward mapping process, the offset coordinates of multiple pixel positions may be mapped to the same target position, meaning the offset coordinates may be identical, resulting in overlapping pixel values. Simultaneously, some target positions may not have any pixel offset coordinates mapped to them, creating gaps in pixel values. For overlapping areas, methods such as averaging pixel values ​​or retaining the maximum value are used for processing. For gap areas, methods such as interpolating neighboring valid pixel values ​​or filling with background values ​​are used to obtain a complete motion-compensated saliency map.

[0043] In addition, for offset coordinates that exceed the boundary of the target saliency map, a boundary copy mode is used for filling. That is, the pixel value of the nearest pixel at the boundary is used as the pixel value of the position that exceeds the range, so as to ensure that the size of the saliency map after motion compensation is consistent with the second saliency map.

[0044] The forward mapping interpolation method disclosed in this embodiment can specifically use the OpenCV remap function to calculate the predicted saliency map. OpenCV is an open-source computer vision library, and the remap function is a specific tool in the OpenCV library used to implement image remapping. Its core task is to move pixels in an image to a new position according to specified rules, thereby generating a completely new image.

[0045] Specifically, the initial pixel coordinate network is created first. and (That is, to determine the initial coordinates corresponding to each pixel position in the second saliency map), with dimensions of (1920, 1080, 1). Each row is [0, 1, ..., 1919]. Each column is [0, 1, ..., 1079]; then, according to Calculate the offset coordinate network (i.e., determine the offset coordinates of each pixel position). , ,at this time, and This indicates which position from the previous frame should be used to sample a pixel in the corresponding direction; then, OpenCV's remap function is called, using the second saliency map. , and Calculate the prediction significance map at time 1 .

[0046] The predicted saliency map is generated by the forward mapping-based interpolation method disclosed in this embodiment. The position of each pixel in the second saliency map is offset by the motion displacement of each pixel position in the dense motion field, which intuitively presents the motion trajectory of the saliency region in the time domain. It is easy to understand and implement, and can make full use of the pixel-by-pixel motion information provided by the dense motion field. Even with the characteristics of sparse texture and high smoothness of the saliency map, it can still obtain relatively accurate prediction results.

[0047] In addition, interpolation can also be performed as follows: The target coordinates of each pixel in the motion-compensated saliency map to be generated are determined. The motion-compensated saliency map to be generated is a saliency map after motion compensation of the second saliency map. Based on each target coordinate and the motion displacement corresponding to the target coordinate in the dense motion field, the source coordinates corresponding to each target coordinate in the second saliency map are determined. If the source coordinates are non-integer coordinates, the number of integer coordinates of the target adjacent to the source coordinates in the second saliency map are determined. Based on the pixel value of each integer coordinate in the number of target integer coordinates, the pixel value of each target coordinate in the motion-compensated saliency map to be generated is determined.

[0048] After obtaining the dense motion field between the first saliency map at the first time step and the second saliency map at the second time step, the motion compensation of the second saliency map is performed using the inverse mapping method disclosed in this embodiment to generate the predicted saliency map at the first time step.

[0049] In the reverse mapping, the motion-compensated saliency map to be generated is first determined. The motion-compensated saliency map is obtained by motion compensation of the second saliency map and is currently in a state before it is generated. The size of the motion-compensated saliency map is the same as that of the first saliency map. In the initial state, all pixel positions have not been assigned values. Then, the target coordinates of each pixel position in the motion-compensated saliency map to be generated are determined. The target coordinates refer to the position of each pixel position in the motion-compensated saliency map in the image coordinate system. They are usually represented by row and column indices or horizontal and vertical coordinates. For example, for an image with height H and width W, its target coordinate range is (0,0) to (H-1, W-1). All target coordinates are traversed, and each position is processed in turn.

[0050] The dense motion field provides inverse motion information from the first saliency map (or the motion-compensated saliency map to be generated) to the second saliency map. For the target coordinates currently being processed, the motion displacement corresponding to the target coordinate position is read from the dense motion field, and the motion displacement is superimposed on the target coordinates to obtain the corresponding source coordinates, which are the coordinates on the second saliency map.

[0051] The source coordinates may be integer coordinates or non-integer coordinates. For integer coordinates, the pixel value at the integer coordinate position in the second saliency map can be directly assigned to the target coordinates without interpolation. For cases where the source coordinates are non-integer coordinates, the pixel value of the target coordinate needs to be estimated based on the pixel values ​​of the neighboring integer coordinates around the source coordinates. Therefore, it is necessary to determine the number of integer coordinates of targets adjacent to the source coordinates in the second saliency map. The number of targets can be determined according to the actual interpolation algorithm used, such as using two adjacent integer coordinates (linear interpolation) or four adjacent integer coordinates (bilinear interpolation). This embodiment does not limit the specific number; any method of estimation using adjacent integer coordinates falls within the protection scope of this solution.

[0052] Based on the pixel value at each integer coordinate position in the integer coordinates of the target quantity, and combined with the relative positional relationship between the source coordinates and these integer coordinates, the estimated pixel value at the source coordinates is calculated by a preset interpolation algorithm, and the estimated pixel value is used as the pixel value at the current target coordinate in the motion-compensated saliency map to be generated.

[0053] Specifically, based on dense sports fields For the frame image at time t-1 (e.g., the second time step), Second saliency plot Motion compensation is performed using an interpolation algorithm (such as bilinear interpolation), with the boundaries copied in a replication mode, thus obtaining the frame image at time t (e.g., the first time step). Prediction significance plot Specifically:

[0054] in, This refers to a pixel position on the frame image at time t-1 (i.e., the second time). The frame image at time t (i.e., the first time) and The corresponding position is calculated as follows: , ; and Corresponding pixels The displacement of motion on the x and y squares in a dense motion field; Let be the predicted saliency data at position (i, j) at time t (where the saliency map is composed of the saliency data of each pixel). The second significance graph The value at the corresponding pixel. , , , The coordinates are respectively , , , ,and ,in, , These are the floor functions for rounding up and rounding down, respectively.

[0055] like Figure 3 The image shown is a schematic diagram illustrating the determination of pixel Q in the predicted saliency map based on the second saliency map. Figure 3 The left side of the middle plot is the second significance plot. The right side shows the prediction significance plot. In the prediction significance plot The prediction significance data for a pixel Q(i,j) is as follows: The predictive significance data It is based on the second significance graph In the pixel Q' ( i ', j The pixel Q' is predicted and determined based on the motion vectors in the dense motion field.

[0056] The calculated coordinates of pixel Q' may not perfectly correspond to the second saliency map. For a specific pixel Q', the coordinates of that pixel might lie in the middle of four other pixels. Bilinear interpolation involves linearly weighting these four pixels to obtain the predicted significance data for pixel Q'. The calculation formula.

[0057] like Figure 4 The diagram illustrates the determination of the predicted significance data for pixel Q in the predicted significance map. Taking pixel Q' as an example, its coordinates are ( i ', j In the horizontal direction, pixel Q' is based on and Horizontal position and The horizontal position is weighted to obtain, that is:

[0058] Q' in the x direction and The distance is less than that with The distance, therefore, The corresponding weights are larger; in the y-direction, temporary values ​​1 and 2 are also weighted in the same way, ultimately yielding... The value of .

[0059] In this embodiment, the second saliency map is interpolated using a dense motion field to generate a predicted saliency map. This predicted saliency map characterizes the state of visual attention that should be presented in the frame image at the first moment under ideal conditions where no unexpected events occur, so as to provide a reliable comparison benchmark for subsequently determining the offset of the quantization parameters.

[0060] The video frame adjustment method disclosed in this embodiment utilizes a dense motion field to perform motion compensation and interpolation on the saliency map at the second moment, generating a predicted saliency map at the first moment. The generated predicted saliency map at the first moment represents the distribution of visual attention in the frame image at the first moment under ideal conditions where no unexpected events occur (such as the appearance of a new object, sudden scene changes, etc.). The predicted saliency map establishes a scientific and reliable comparison benchmark for judging whether an unexpected event has occurred, so that when comparing the first saliency map at the first moment with the predicted saliency map in subsequent moments, the offset of the first saliency map relative to the predicted saliency map can be accurately determined, thereby ensuring accurate adjustment of the bitrate of the frame image in the video.

[0061] Furthermore, the video frame adjustment method disclosed in this embodiment is implemented using a saliency map, which is generated by a model. Specifically: The video is input into a pre-trained target model to obtain the saliency map corresponding to each frame image in the video output by the target model.

[0062] The saliency map can be calculated using a trained neural network model (i.e., the target model) to ensure high accuracy. For example, the target model can be pre-trained using the MobileNetV2 network. During inference, the source video is input into the target model to obtain the ".roi" file output by the target model. This file records the saliency map of each frame in a specific file format.

[0063] Furthermore, after obtaining the saliency map, it is necessary to normalize the saliency map, that is, to normalize the pixel value of each pixel in the first saliency map and the second saliency map.

[0064] In the saliency map, each pixel value is a value in the range of 0-255. To optimize subsequent calculations and prevent inconsistencies in calculation results due to pixel value size, normalization processing is required. For example, if a pixel is 10 bits, its maximum pixel value is 2^10 minus 1, which is 1023. If normalization is not performed, the calculation results will be abnormal.

[0065] Normalization is the process of normalizing the value of each pixel to the range of 0-1. For example, if the original pixel value is 0, the normalized pixel value will still be 0. If the original pixel value is 128, the normalized pixel value will be 128 / 255 = 0.501961. If the original pixel value is 255, the normalized pixel value will be 255 / 255 = 1.

[0066] In addition, the resolution of the saliency map is the same as that of the source video, such as 1920×1080.

[0067] This embodiment discloses a video frame adjustment method, the flowchart of which is as follows: Figure 5 As shown, it includes: Step S51: Obtain the first saliency map corresponding to the frame image at the first moment in the video and the second saliency map corresponding to the frame image at the second moment, wherein the second moment is the moment before the first moment and adjacent to the first moment; Step S52: Determine the dense motion field between the first saliency map and the second saliency map; Step S53: Determine the predicted saliency map corresponding to the frame image at the first moment based on the dense motion field and the second saliency map; Step S54: Determine the target temporal weight of the frame image at the first time based on the first saliency map and the predicted saliency map at the first time. Step S55: Utilize the correspondence between temporal weights and offsets to determine the offset of the quantization parameters of the frame image at the first moment corresponding to the target temporal weights. Step S56: Adjust the bitrate of the frame image at the first moment based on the offset of the quantization parameters of the frame image at the first moment.

[0068] The process involves obtaining saliency maps for two adjacent frames in the video, specifically the first saliency map corresponding to the frame at the first time step and the second saliency map corresponding to the frame at the second time step (the second time step is earlier than the first time step). A dense motion field between the first and second saliency maps is determined, and a predicted saliency map corresponding to the frame at the first time step is calculated based on this dense motion field and the second saliency map. This allows for the determination of the quantization parameter offset for the frame at the first time step, based on both the first and predicted saliency maps, and the bitrate of the frame at the first time step is adjusted accordingly. Establishing a temporal correlation through the dense motion field between saliency maps generates the predicted saliency map. This process simulates the temporal continuity of visual attention, providing a more accurate benchmark for evaluating the difference between the actual and predicted saliency maps, ensuring the accuracy of bitrate adjustment, and optimizing the visual experience of the video.

[0069] Specifically, determining the offset of the quantization parameters of the frame image at the first time step based on the first saliency map and the predicted saliency map at the first time step includes: determining the target temporal weight of the frame image at the first time step based on the first saliency map and the predicted saliency map at the first time step; and determining the offset of the quantization parameters of the frame image at the first time step corresponding to the target temporal weight by using the correspondence between the target temporal weight and the offset.

[0070] Target temporal weight is a numerical metric used to quantify the visual importance of each frame in a video to the human eye. Specifically, it represents the deviation between the actual saliency distribution of the frame and the saliency distribution predicted based on historical motion trends (i.e., visual unexpectedness / surprise). Its core function is to guide bitrate allocation. The larger the target temporal weight, the more unexpected the visual changes in the current frame (e.g., shot transitions, new object entry, etc.). In this case, the frame is more sensitive to the human eye and requires more bitrate resources to ensure its quality. The smaller the target temporal weight, the smoother the temporal changes (e.g., stillness or uniform motion). In this case, the human eye has a high tolerance for distortion, and bitrate can be appropriately saved.

[0071] After obtaining the first saliency map and the predicted saliency map at the first time step, the two maps can be analyzed to determine the difference between them. This difference is then used to determine the target temporal weights. The difference between the first and predicted saliency maps can be determined by comparing each pixel in both maps to identify the degree of deviation (i.e., the difference). A larger difference indicates a more significant temporal abrupt change. The difference can be determined using absolute error and SAD or mean squared error (MSE). After obtaining this difference, it can be mapped to frame-level temporal weights, i.e., the target temporal weights, making them positively correlated with perceptual importance.

[0072] In this embodiment, the target temporal weight is determined based on the first saliency map at the first moment and the predicted saliency map at the first moment. The process relies only on the saliency map and does not require the original video to participate, which reduces the computational complexity and system coupling. In addition, the difference between the actual saliency map and the predicted saliency map is used to accurately capture temporal abrupt changes such as shot switching and the appearance of new objects, so that the bitrate allocation meets the visual perception requirements.

[0073] After determining the target temporal weight at the first moment, the offset of the quantization parameters of the frame image at the first moment corresponding to the target temporal weight is determined by combining the correspondence between the target temporal weight and the offset.

[0074] A correspondence between target temporal weights and offsets is established in advance. Then, based on this correspondence, the offset corresponding to the target temporal weights is determined so that the offset is used as the offset of the quantization parameters of the frame image at the first moment. The offset corresponding to high target temporal weights is small, and the offset corresponding to low temporal weights is large.

[0075] Specifically, after obtaining the target temporal weights, they can first be normalized, that is, normalized from [min, max] to the range [0, 1]. Considering the range of the quantization parameter QP in the encoder, such as the range of the quantization parameter of a certain encoder being [0, 255], the normalized target temporal weights [0, 1] can be mapped to the quantization parameter offsets [4, -12]. That is, the offset corresponding to the lowest target temporal weight is +4, and the offset corresponding to the highest target temporal weight is -12. Thus, the correspondence between the target temporal weights and the offsets can be fitted, which can be specifically as follows: ,in, This is the offset. As the target time domain weight, input any target time domain weight into this correspondence to determine the offset corresponding to that target time domain weight.

[0076] Furthermore, in this embodiment, when obtaining the target temporal weight, the specific steps are as follows: determine the initial temporal weight of the frame image at the first moment based on the first saliency map, the second saliency map, and the predicted saliency map; and smooth the initial temporal weight to obtain the target temporal weight of the frame image at the first moment.

[0077] That is, firstly, the initial temporal weights of the frame image at the first moment are obtained, and then the initial temporal weights are smoothed to obtain the target temporal weights.

[0078] Specifically, an exponential moving average can be used to adjust the initial time-domain weights. Perform time-domain smoothing, i.e.: ,in, The target time-domain weights at time t (e.g., the first time). λ represents the target time-domain weight at time t-1 (e.g., the second time), and λ is the smoothing coefficient, which can be 0.7.

[0079] It should be noted that if the frame image at time t is the first frame image, meaning there is no image at time t-1, then during smoothing, the formula... Can be equal to .

[0080] By smoothing the initial time-domain weights, the target time-domain weights are obtained, thus avoiding excessive fluctuations in the time-domain weights that could affect subjective perception.

[0081] Furthermore, obtaining the initial time-domain weights can be specifically as follows: A first value is determined based on the pixel values ​​at the target location in the first saliency map and the pixel values ​​at the target location in the predicted saliency map. The first value is used to characterize the difference between the first saliency map and the predicted saliency map. A second value is determined based on the first saliency map and the second saliency map. The second value is used to characterize the change between the attention value of the first saliency map and the attention value of the second saliency map. The initial temporal weights of the frame image at the first time step are determined based on at least the first value and the second value.

[0082] The first numerical value represents the difference between the first saliency map and the predicted saliency map. It is actually the mean of the absolute prediction residuals of the first saliency map and the second saliency map. Its physical meaning is the difference between the actual attention and the predicted attention. It is used to describe the abrupt change in the attention of the image frame at the first time. The average value is used to measure the overall difference of an image frame.

[0083] Based on this, the first value can be determined as follows: The difference between each pixel position in the first saliency map and the pixel value at the corresponding pixel position in the predicted saliency map is determined to obtain the prediction residual for each pixel position; the absolute value of the prediction residual for each pixel position is averaged, and the average value is determined as the first value.

[0084] The first value needs to be determined based on the pixel values ​​at the corresponding positions in the first saliency map and the predicted saliency map, that is: Where K1 is the first value, and M and N are the width and height of the first saliency plot, respectively. and These are the first significance plot and the predicted significance plot, respectively. i, j The pixel value at position K1 represents the degree of surprise of the first saliency map of the current frame image relative to the predicted saliency map. The larger K1 is, the more significant the temporal abrupt change in attention occurs.

[0085] The second value represents the change between the attention values ​​of the first and second saliency maps. It is the DC component difference between the two maps, indicating the rate and direction of change of the saliency mean, reflecting the dynamic trend of attention shift. The sign of the second value indicates the direction of change of the saliency mean, and the magnitude of the value indicates the rate of change. For example, as a face moves closer to the camera, the area it occupies on the image increases, and correspondingly, the attention area also increases. Therefore, the mean of the saliency map also increases, and K2 becomes positive, indicating that the area of ​​interest to the human eye has expanded.

[0086] Based on this, the determination of the second value can be as follows: Determine the average value of all pixels in the first saliency map and the average value of all pixels in the second saliency map; determine the second value based on the difference between the average value of all pixels in the first saliency map and the average value of all pixels in the second saliency map.

[0087] The second value needs to be determined based on the first significance plot and the second significance plot, that is... K2 is the second value. In the first significance graph ( i, j The pixel value at the given location is used to calculate the overall attention level of the frame image by averaging the values ​​at that location.

[0088] Since the target temporal weight is a numerical indicator used to quantify the visual importance of each frame in a video to the human eye, the initial temporal weight is the same as the target temporal weight, and is also a numerical indicator used to quantify the visual importance of each frame in a video to the human eye. Therefore, after determining the first value representing the difference between the actual attention and the predicted attention at the first moment, and the second value representing the change between the attention value of the saliency map at the first moment and the attention value of the saliency map at the second moment, the indicator of the visual importance of the frame image at the first moment can be determined at least based on the first value and the second value, that is, the initial temporal weight of the frame image at the first moment.

[0089] Specifically, determining the initial temporal weights of the frame image at the first moment based at least on the first and second values ​​can be achieved by: determining the average pixel value of all pixels included in the first saliency map; determining the weight coefficients corresponding to the average value, the first value, and the second value; and summing the average value, the first value, and the second value according to their respective weight coefficients to obtain the initial temporal weights of the frame image at the first moment.

[0090] The initial temporal weights of the frame image at the first time step are obtained by weighted summing of the average value, the first value, and the second value. The average value is the average pixel value of all pixels included in the first saliency map. Where K3 is the average value, then the initial time-domain weights are... for: ,in, α + β + γ ≈1, α The weighting factor is the average pixel value of all pixels in the first saliency map. β The weighting coefficient for the first value. γ The weighting coefficient for the second value. Furthermore, β can be specifically obtained from β1 × β2, and correspondingly, γ can be obtained from γ1 × γ2, such as: α = 0.55, β1 = 0.25, β2 = 1.5, γ1 = 0.15, γ2 = 1.2. Here, β2 and γ2 are used to enhance the corresponding subjective perception characteristics (i.e., the first and second values), and are values ​​set for the original weight of the corresponding characteristic. For example, for K1, the original weight of K1 is increased by 1.5 times, i.e., β1 is the original weight, and β2 is the factor by which the original weight β1 is increased.

[0091] In this embodiment, by introducing joint modeling of salience surprise (K1) and salience temporal momentum (K2), more accurate bit priority allocation can be obtained in abrupt frames such as shot switching, the appearance of new objects, and rapid shifts in attention, thus optimizing the subjective experience.

[0092] The video frame adjustment method disclosed in this embodiment obtains the first saliency map and the second saliency map corresponding to the frame images of adjacent first and second moments, respectively. This provides the core data input for subsequent analysis, allowing the analysis to focus on the saliency map rather than the original video frame, significantly reducing the complexity of data processing. It achieves preliminary decoupling between perceptual analysis and video coding, making the computation more efficient and flexible. The method also determines the dense motion field between the first and second saliency maps, and through calculation of the dense motion field, achieves more precise capture of the complete motion trajectory of the saliency region from the previous frame to the current frame. Based on the dense motion... The field and the second saliency map determine the predicted saliency map at the first time step, so that the predicted saliency map at the first time step can be compared with the first saliency map at the first time step to determine the target temporal weight at the first time step. By overcoming the shortcomings of the predicted saliency map, a reliable comparison benchmark is provided for subsequent difference analysis. The target temporal weight is obtained by comparing the actual saliency map and the predicted saliency map, which directly quantizes the visual difference. The target temporal weight is mapped to the offset of the quantization parameter. The temporal weight is converted into the quantization parameter offset that the encoder can directly understand and execute through a preset mapping relationship, thereby realizing the precise adjustment of the bit rate.

[0093] Furthermore, in the video frame adjustment method disclosed in this embodiment, the offset of the quantization parameters of the frame image at the first moment corresponding to the target temporal weight is determined by utilizing the correspondence between the target temporal weight and the offset. Specifically, this can be achieved as follows: The target temporal weights of each frame in the video are determined to obtain the target temporal weight sequence of the video. Using the correspondence between the target temporal weights and the offsets, the offset of the quantization parameters of each frame in the video corresponding to the target temporal weights is determined.

[0094] The target temporal weights of each frame in the video are determined. Only after the target temporal weights of each frame in the video are determined will the subsequent offsets be determined so that the bitrate of each frame in the video can be adjusted simultaneously.

[0095] By obtaining the target temporal weights of each frame in the video, a target temporal weight sequence can be obtained. Then, the offsets and bitrates are uniformly determined based on this target temporal weight sequence. This allows for precise calculation of the weight distribution across the entire video, ensuring that the total quantization parameter values ​​of all frames remain constant, thus guaranteeing a consistent total bitrate from an algorithm design perspective. Determining the offsets of each frame based on the target temporal weight sequence ensures that the keyframe with the highest weight is identified and assigned the largest negative offset, while the keyframe with the lowest weight is identified and assigned the largest positive offset, ensuring more reasonable resource allocation. Furthermore, batch-determining the offsets and adjusting the bitrates of each frame in the video using the target temporal weight sequence smooths the global strategy and avoids adjustment anomalies caused by local information.

[0096] This embodiment discloses a video frame adjustment system, the structural schematic diagram of which is shown below. Figure 6 As shown, it includes: The system includes a obtaining unit 61, a first determining unit 62, a second determining unit 63, a third determining unit 64, and an adjusting unit 65.

[0097] The obtaining unit 61 is used to obtain a first saliency map corresponding to a frame image at a first moment in the video and a second saliency map corresponding to a frame image at a second moment, wherein the second moment is a moment before the first moment and adjacent to the first moment; The first determining unit 62 is used to determine the dense motion field between the first saliency map and the second saliency map; The second determining unit 63 is used to determine the predicted saliency map corresponding to the frame image at the first moment based on the dense motion field and the second saliency map. The third determining unit 64 is used to determine the offset of the quantization parameters of the frame image at the first time based on the first saliency map and the predicted saliency map at the first time. The adjustment unit 65 is used to adjust the bitrate of the frame image at the first moment based on the offset of the quantization parameters of the frame image at the first moment.

[0098] The video frame adjustment system disclosed in this embodiment is implemented based on the video frame adjustment method disclosed in the above embodiment, and will not be described again here.

[0099] The video frame adjustment system disclosed in this embodiment obtains saliency maps corresponding to two adjacent frames in a video. The saliency map corresponding to the previous frame is determined as the second saliency map, and the saliency map corresponding to the next frame is determined as the first saliency map. Based on the dense motion field between the first and second saliency maps and the second saliency map, a predicted saliency map for the next frame is determined. Based on the predicted saliency map and the first saliency map, the offset of the quantization parameters for the next frame is determined, and the bitrate of the next frame is adjusted based on this offset. This scheme establishes a temporal correlation through the dense motion field between saliency maps, thereby generating a predicted saliency map. This process simulates the temporal continuity of visual attention, providing a more accurate benchmark for evaluating the difference between the actual and predicted saliency maps. By analyzing the actual and predicted saliency maps of a frame image, the offset of the quantization parameters for that frame image is determined, quantifying the deviation between the actual and predicted distribution of attention in the current frame image. Based on this, the bitrate is adjusted, optimizing the visual experience.

[0100] This embodiment discloses an electronic device, the structural schematic diagram of which is shown below. Figure 7 As shown, it includes: Processor 71 and memory 72.

[0101] The processor 71 is configured to obtain a first saliency map corresponding to a frame image at a first moment in the video and a second saliency map corresponding to a frame image at a second moment, wherein the second moment is a moment before and adjacent to the first moment; determine a dense motion field between the first and second saliency maps; determine a predicted saliency map corresponding to the frame image at the first moment based on the dense motion field and the second saliency map; determine the offset of the quantization parameters of the frame image at the first moment based on the first and predicted saliency maps; and adjust the bitrate of the frame image at the first moment based on the offset of the quantization parameters of the frame image at the first moment. The memory 72 is used to store the program required by the processor to perform the above-mentioned processing.

[0102] The electronic device disclosed in this embodiment is implemented based on the video frame adjustment method disclosed in the above embodiment, and will not be described again here.

[0103] The electronic device disclosed in this embodiment obtains saliency maps corresponding to two adjacent frames in a video. The saliency map corresponding to the previous frame is determined as the second saliency map, and the saliency map corresponding to the next frame is determined as the first saliency map. Based on the dense motion field between the first and second saliency maps and the second saliency map, a predicted saliency map for the next frame is determined. Furthermore, based on the predicted saliency map and the first saliency map, the offset of the quantization parameters for the next frame is determined, and the bitrate of the next frame is adjusted based on this offset. This scheme establishes a temporal correlation through the dense motion field between saliency maps, thereby generating a predicted saliency map. This process simulates the temporal continuity of visual attention, providing a more accurate benchmark for evaluating the difference between the actual and predicted saliency maps. By analyzing the actual and predicted saliency maps of a frame image, the offset of the quantization parameters for that frame image is determined, quantifying the deviation between the actual and predicted distribution of attention in the current frame image. Based on this, the bitrate is adjusted, optimizing the visual experience.

[0104] This application embodiment also provides a readable storage medium storing a computer program thereon. The computer program is loaded and executed by a processor to implement the steps of the above-described video frame adjustment method. The specific implementation process can be referred to the description of the corresponding part of the above embodiment, and will not be repeated in this embodiment.

[0105] This application also proposes a computer program product or computer program including computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations of the video frame adjustment method described above. Specific implementation processes can be referred to the descriptions of the corresponding embodiments above, and will not be repeated here.

[0106] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0108] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0109] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A video frame adjustment method, characterized in that, include: Obtain a first saliency map corresponding to the frame image at the first moment in the video and a second saliency map corresponding to the frame image at the second moment, wherein the second moment is the moment before the first moment and adjacent to the first moment; Determine the dense motion field between the first saliency map and the second saliency map; Based on the dense motion field and the second saliency map, the predicted saliency map corresponding to the frame image at the first moment is determined; The offset of the quantization parameters of the frame image at the first time moment is determined based on the first saliency map at the first time moment and the predicted saliency map. The bitrate of the frame image at the first moment is adjusted based on the offset of the quantization parameters of the frame image at the first moment.

2. The method according to claim 1, characterized in that, The step of determining the predicted saliency map corresponding to the frame image at the first moment based on the dense motion field and the second saliency map includes: According to the dense motion field, the second saliency map is interpolated to obtain a motion-compensated saliency map, and the motion-compensated saliency map is determined as the predicted saliency map corresponding to the frame image at the first time.

3. The method according to claim 2, characterized in that, The step of interpolating the second saliency map according to the dense motion field to obtain a motion-compensated saliency map includes: Determine the initial coordinates corresponding to each pixel position in the second saliency map; Based on the motion displacement of each pixel position in the dense motion field and the initial coordinates of the corresponding pixel, the offset coordinates of each pixel position are determined. The motion-compensated saliency map is determined based on the offset coordinates of each pixel position.

4. The method according to claim 2, characterized in that, The step of interpolating the second saliency map according to the dense motion field to obtain a motion-compensated saliency map includes: Determine the target coordinates of each pixel position in the motion-compensated saliency map to be generated, wherein the motion-compensated saliency map to be generated is a saliency map after motion compensation of the second saliency map; Based on each target coordinate and the motion displacement corresponding to the target coordinate in the dense motion field, determine the source coordinates corresponding to each target coordinate in the second saliency map; If the source coordinates are non-integer coordinates, determine the number of integer coordinates of targets adjacent to the source coordinates in the second saliency map; Based on the pixel value at each integer coordinate in the integer coordinates of the target quantity, determine the pixel value at each target coordinate in the motion-compensated saliency map to be generated.

5. The method according to claim 1, characterized in that, The step of determining the offset of the quantization parameters of the frame image at the first time step based on the first saliency map at the first time step and the predicted saliency map includes: The target temporal weights of the frame images at the first time point are determined based on the first saliency map at the first time point and the predicted saliency map. By utilizing the correspondence between the target temporal weight and the offset, the offset of the quantization parameters of the frame image at the first moment corresponding to the target temporal weight is determined.

6. The method according to claim 5, characterized in that, The step of determining the target temporal weight of the frame image at the first time step based on the first saliency map at the first time step and the predicted saliency map includes: A first value is determined based on the pixel value at the target location in the first saliency map and the pixel value at the target location in the predicted saliency map. The first value is used to characterize the difference between the first saliency map and the predicted saliency map. A second value is determined based on the first saliency map and the second saliency map. The second value is used to characterize the amount of change between the attention value of the first saliency map and the attention value of the second saliency map. The target temporal weight of the frame image at the first moment is determined based at least on the first value and the second value.

7. The method according to claim 6, characterized in that, Determining the first value based on the pixel value at the target location in the first saliency map and the pixel value at the target location in the predicted saliency map includes: The difference between each pixel position in the first saliency map and the pixel value at the corresponding pixel position in the predicted saliency map is determined to obtain the prediction residual for each pixel position; The absolute value of the prediction residual for each pixel position is averaged, and the resulting average value is determined as the first value.

8. The method according to claim 6, characterized in that, The step of determining the second value based on the first saliency map and the second saliency map includes: Determine the average value of all pixels in the first saliency map and the average value of all pixels in the second saliency map; The second value is determined based on the difference between the average value of all pixels in the first saliency map and the average value of all pixels in the second saliency map.

9. The method according to claim 5, characterized in that, The step of determining the offset of the quantization parameters of the frame image at the first moment corresponding to the target temporal weight by utilizing the correspondence between the target temporal weight and the offset includes: Determine the target temporal weight of each frame in the video to obtain the target temporal weight sequence of the video; By utilizing the correspondence between the target temporal weight and the offset, the offset of the quantization parameter of each frame image in the video corresponding to the target temporal weight is determined.

10. An electronic device, characterized in that, include: A processor is configured to obtain a first saliency map corresponding to a frame image at a first moment in a video and a second saliency map corresponding to a frame image at a second moment, wherein the second moment is a moment prior to and adjacent to the first moment; determine a dense motion field between the first saliency map and the second saliency map; determine a predicted saliency map corresponding to the frame image at the first moment based on the dense motion field and the second saliency map; determine an offset of the quantization parameters of the frame image at the first moment based on the first saliency map and the predicted saliency map; and adjust the bitrate of the frame image at the first moment based on the offset of the quantization parameters of the frame image at the first moment. The memory is used to store the programs required by the processor to perform the above-described processing.

Images