Method for encoding and decoding multi-view video, encoder and decoder

Abstract

This application provides a multi-view video encoding method, decoding method, encoder, and decoder. By establishing a viewpoint correspondence between a first and a second synchronously acquired video stream, and determining the motion vector between the two video streams in conjunction with the geometric parameters of the acquisition device, a more targeted reference position can be provided for the current encoding block in the second video stream within the corresponding reference frame. Furthermore, by determining reference pixels around the corresponding reference block and completing the encoding and decoding of the current encoding block, the amount of reference data read / write and computation caused by large-scale blind searching is reduced. While ensuring prediction accuracy, the overhead of on-chip storage and external storage access is reduced, compression efficiency is improved, and the hardware implementation burden is reduced.

Description

Technical Field

[0001] This application relates to the field of video coding, and in particular to a method for encoding, decoding, encoder and decoder of multi-view video. Background Technology

[0002] With the widespread application of smart wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, and binocular camera display devices, the demand for multi-view video acquisition, transmission, and display is increasing. In the process of acquiring and transmitting multi-view video, inter-frame predictive coding is typically performed on multiple video streams, and motion estimation methods such as block matching are used to achieve data compression.

[0003] However, existing solutions in binocular parallax scenarios often use general motion search strategies, which often require a large search range to complete the matching, resulting in an increase in the amount of reference data read / write and computation. When the shooting distance or focal length changes, the reference position estimation is prone to deviation, which further reduces the compression efficiency and increases the storage bandwidth, power consumption and hardware implementation burden of mobile devices.

[0004] Therefore, how to balance prediction accuracy, compression efficiency, and hardware resource consumption under changing scenes in binocular video coding has become a technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a method for encoding, decoding, encoder, and decoder of multi-view video to solve the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide a multi-view video encoding method, applied to an encoder; the method includes:

[0007] Acquire the first and second video streams synchronously acquired by the acquisition device. The video frames in the second video stream have corresponding reference frames in the first video stream.

[0008] Based on the geometric parameters of the acquisition device, determine the motion vector between the second video stream and the first video stream;

[0009] For the current coded block of any video frame in the second video, based on the motion vector, determine the corresponding reference block in the reference frame corresponding to the targeted video frame, set the motion search window with the reference block as the center, and determine the reference pixel;

[0010] The encoding result of the current encoding block is determined based on the reference pixels.

[0011] In one possible implementation, determining the motion vector between the second video stream and the first video stream based on the geometric parameters of the acquisition device includes:

[0012] Acquire the focal length, image distance, baseline distance, sensor width, and video resolution width of the acquisition device;

[0013] Determine the resolution focal length based on focal length, sensor width, and video resolution width;

[0014] Based on the focal length and image distance, determine the object distance between the imaging object and the acquisition device;

[0015] The motion vector is determined based on the resolution focal length, object distance, and baseline distance.

[0016] In one possible implementation, the motion vector is determined based on the resolution focal length, object distance, and baseline distance, including:

[0017] Monitor the dynamic changes in the scene and adjust the weighting coefficients of resolution focal length, object distance, and baseline distance based on the monitoring results;

[0018] Based on the adjusted weighting coefficients, the motion vector is determined according to the resolution focal length, object distance, and baseline distance.

[0019] In one possible implementation, the method further includes:

[0020] Based on the monitoring results, the size of the motion search window is adjusted, and the reference pixel is determined according to the adjusted motion search window.

[0021] In one possible implementation, based on the motion vector, a corresponding reference block is determined in the reference frame corresponding to the target video frame; a motion search window is set with the reference block as the center; and reference pixels are determined, including:

[0022] Based on the horizontal and vertical components of the motion vector, the region in the reference frame corresponding to the offset of the current coded block position is determined, and this region is determined as the reference block;

[0023] The geometric center of the reference block is used as the starting point for the search;

[0024] Based on the preset search range threshold, a region centered on the search starting point is defined in the reference frame as the motion search window;

[0025] Pixel matching is performed within the motion search window, and the pixel in the pixel block with the lowest matching cost is determined as the reference pixel corresponding to the current coding block.

[0026] In one possible implementation, the preset search range threshold is dynamically set based on the confidence level of the motion vector and / or the size of the current coded block.

[0027] When the confidence level of the motion vector is less than the first threshold, a first search range is set;

[0028] When the confidence level of the motion vector is greater than or equal to the first threshold, a second search range greater than the first search range is set.

[0029] The search range threshold is positively correlated with the size of the current coded block.

[0030] In one possible implementation, the encoder is integrated into an encoding chip, which also includes an on-chip buffer; after setting the motion search window centered on the reference block, and before determining the reference pixel, the following steps are also included:

[0031] The data read address is generated based on the pixel range covered by the motion search window in the reference frame;

[0032] Based on the data read address, pixel data within a pixel range is read from the external memory of the encoding chip and loaded into the on-chip cache;

[0033] When performing pixel matching within the motion search window, pixel data is read from the on-chip cache.

[0034] In one possible implementation, when the reference frame includes a forward reference frame and a backward reference frame:

[0035] Based on the motion vector, the first sub-reference block is determined in the forward reference frame and the first sub-search window is set; the second sub-reference block is determined in the backward reference frame and the second sub-search window is set.

[0036] Within the motion search window, pixel matching is performed, and the pixels in the pixel block with the lowest matching cost are determined as the reference pixels for the current coding block, including:

[0037] Matching is performed in the first sub-search window and the second sub-search window respectively to obtain the first sub-pixel block and the second sub-pixel block;

[0038] The pixel with the smaller matching cost in the first and second sub-pixel blocks is selected as the final reference pixel; or the pixels in the first and second sub-pixel blocks are averaged to obtain the reference pixel.

[0039] Secondly, embodiments of this application provide a multi-view video decoding method, applied to a decoder, the method comprising:

[0040] The encoding results of the first and second video streams are received, as well as the motion vector between the second and first video streams. The video frames in the second video stream have corresponding reference frames in the first video stream.

[0041] For the current decoding block of any video frame in the second video, based on the motion vector, determine the corresponding reference block in the reference frame corresponding to the targeted video frame, set the motion search window with the reference block as the center, and determine the reference pixel;

[0042] Based on the decoding result of the reference pixel, the encoding result of the current decoding block is decoded to obtain the decoding result of the current decoding block.

[0043] Thirdly, embodiments of this application provide an encoder for performing a multi-view video encoding method as described in the first aspect and / or possible implementations of the first aspect.

[0044] Fourthly, embodiments of this application provide a decoder for performing the multi-view video decoding method as mentioned in the second aspect.

[0045] Fifthly, embodiments of this application provide a multi-view video encoding and decoding apparatus, including an encoder and a decoder, wherein the encoder includes:

[0046] The data acquisition module is used to acquire the first and second video streams synchronously acquired by the acquisition device. The video frames in the second video stream have corresponding reference frames in the first video stream.

[0047] The vector determination module is used to determine the motion vector between the second video and the first video based on the geometric parameters of the acquisition device.

[0048] The search and matching module is used to determine the corresponding reference block in the reference frame corresponding to the target video frame based on the motion vector for the current encoded block of any video frame in the second video, and to set the motion search window with the reference block as the center to determine the reference pixel.

[0049] The encoding module is used to determine the encoding result of the current encoding block based on the reference pixels;

[0050] The decoder includes:

[0051] The decoding module is used to decode the current coding block based on the encoding result of the corresponding reference block in the reference frame and the encoding result of the current coding block, so as to obtain the decoding result of the current coding block.

[0052] This application provides a multi-view video encoding method, decoding method, encoder, and decoder. By establishing a viewpoint correspondence between a first and a second synchronously acquired video stream, and determining the motion vector between the two video streams in conjunction with the geometric parameters of the acquisition device, a more targeted reference position can be provided for the current encoding block in the second video stream within the corresponding reference frame. Furthermore, by determining reference pixels around the corresponding reference block and completing the encoding and decoding of the current encoding block, the amount of reference data read / write and computation caused by large-scale blind searching is reduced. While ensuring prediction accuracy, the overhead of on-chip storage and external storage access is reduced, compression efficiency is improved, and the hardware implementation burden is reduced. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0054] Figure 1 Flowchart of the multi-view video encoding method provided in this application Figure 1 ;

[0055] Figure 2 A schematic diagram of the motion search window provided in this application;

[0056] Figure 3 A flowchart illustrating the multi-view video decoding method provided in this application;

[0057] Figure 4 Flowchart of the multi-view video encoding method provided in this application Figure 2 ;

[0058] Figure 5 This is a schematic diagram of the optical path of the binocular camera provided in this application;

[0059] Figure 6 Flowchart of the multi-view video encoding method provided in this application Figure 3 ;

[0060] Figure 7 This is a schematic diagram of the encoding provided in this application;

[0061] Figure 8 A schematic diagram of the structure of the multi-view video encoding and decoding device provided in this application.

[0062] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0064] This technology belongs to the field of multi-view video encoding and decoding, and is applicable to binocular cameras, AR (Augmented Reality) or VR (Virtual Reality) terminals, smart wearable devices, and other imaging systems capable of simultaneously acquiring two video streams. Such systems typically consist of an acquisition device, an encoder, a transmission link, and a decoder. The acquisition device simultaneously outputs the first and second video streams, which are then encoded for local storage, wireless transmission, or playback on a display.

[0065] Existing stereo video processing schemes typically use one of the two video streams as a reference to perform inter-frame predictive coding on the other. The basic approach is to search for a matching block in the corresponding reference frame of the first video stream while encoding the current coding block of the second video stream. Then, based on the matching result, prediction information and encoding results are generated, thereby compressing the dual-stream video data.

[0066] However, in binocular parallax scenarios, there is an overall displacement between the left and right viewpoints caused by the shooting geometry. If a general motion search strategy is still used, a large search range is often required to cover possible matching positions, resulting in a significant increase in reference pixel readouts, on-chip cache usage, and external storage access. Especially when the shooting distance changes, the focus is adjusted, or when observing close-range targets, the reference position is more likely to deviate from the true parallax region, which in turn reduces matching accuracy, increases residuals, causes higher bitrates and power consumption, and increases the burden on mobile hardware implementation.

[0067] Therefore, balancing prediction accuracy, compression efficiency, and hardware resource consumption under changing scene conditions in binocular video coding has become a pressing technical problem. To address this issue, a multi-view video encoding and decoding method is proposed. On the encoder side, a first and second video stream are acquired simultaneously by the acquisition device. Motion vectors between the two video streams are determined based on the geometric parameters of the acquisition device. For the current coding block of the second video stream, a reference block is determined in the reference frame based on this motion vector. A motion search window is then set centered on the reference block, and the encoding result of the current coding block is determined using reference pixels within the window. On the decoder side, decoding is completed based on the encoding results of the reference block and the current coding block. By combining the processing architecture of the acquisition device, encoder, transmission link, and decoder, this approach can reduce the storage access and computational overhead caused by global search.

[0068] In one possible implementation, such as Figure 1 As shown, the encoding methods for multi-view video include:

[0069] Step 101: Acquire the first and second video streams synchronously acquired by the acquisition device. The video frames in the second video stream have corresponding reference frames in the first video stream.

[0070] In this application, the first video stream and the second video stream are two video data streams acquired by the acquisition device under synchronous conditions. Synchronous acquisition means that the two video streams have a corresponding relationship on the time axis, that is, the video frames in the second video stream can find a reference frame formed at the same acquisition time in the first video stream. A video frame can be a complete image data, the current coding block can be a processing unit divided from the video frame, and the reference frame refers to the frame data used to provide the basis for pixel prediction when performing predictive coding or predictive decoding on the current video frame.

[0071] In this step, the encoder acquires two video data streams and establishes a correspondence between the first and second video frame sequences. For each pair of synchronized video frames, the encoder uses the video frames from the first video stream as candidate reference frames and the second video stream as the source of the current frame in the video stream to be encoded. After this step, the encoder obtains two temporally corresponding video data streams, providing an input basis for subsequently determining the encoding reference position of the second video stream based on the first video stream. Based on the above analysis, it can be seen that establishing the correspondence between synchronized acquisition and reference frames enables cross-viewpoint prediction to be based on the scene content at the same moment, making subsequent reference management more certain.

[0072] Step 102: Determine the motion vector between the second video stream and the first video stream based on the geometric parameters of the acquisition device.

[0073] In this application, geometric parameters are used to reflect the spatial relative relationship between two acquisition viewpoints, including at least one or more of the following: binocular baseline length, relative installation position of the two cameras, optical axis angle, focal length, principal point position, and relative offset on the imaging plane.

[0074] In this application, motion vectors are used to characterize the displacement relationship when the corresponding region of the second video path is mapped to the reference frame of the first video path. The motion vector can be a two-dimensional displacement with integer pixel precision or a two-dimensional displacement containing fractional pixel components, such as a horizontal component (dx) and a vertical component (dy). The encoder can read calibration parameters during device initialization and establish a spatial correspondence model from the second viewpoint to the first viewpoint before video encoding begins. In one possible embodiment, for a calibrated binocular system, the main displacement between the two viewpoints can be concentrated in the horizontal direction. In this case, disparity can be determined based on relevant geometric parameters and mapped to motion vectors. For acquisition devices with installation deflection, the encoder can also determine candidate reference positions based on the corresponding geometric relationships, and then generate motion vectors from the difference between the current encoding block coordinates and the candidate reference positions. The encoder writes the obtained motion vectors into the cross-viewpoint prediction context of the current frame and calls them during block-level encoding.

[0075] In this application, the geometric parameters in the technical terms are not abstract descriptions, but specific calibrated quantities involved in displacement calculation; the motion vector is not limited to the object motion in traditional time-domain motion estimation, but includes cross-viewpoint position offset caused by binocular shooting geometry.

[0076] Step 103: For the current coding block of any video frame in the second video, determine the corresponding reference block in the reference frame corresponding to the targeted video frame based on the motion vector, set the motion search window with the reference block as the center, and determine the reference pixel.

[0077] The encoder performs block-level processing on the current video frame in the second video stream according to a predetermined scanning order. The block size of the current coded block can be a fixed-size block or a non-fixed-size block after division. The encoder first obtains the block coordinates of the current coded block in the second video stream frame, then applies the motion vector obtained in step 102 to these block coordinates, and determines a candidate reference position in the corresponding first video stream reference frame. Using this candidate reference position as a reference, a region corresponding to the current coded block is extracted and used as a reference block. Then, a motion search window is established centered on the reference block. A reference pixel refers to a pixel determined within the motion search window that is used for prediction processing of the current coded block. The encoder obtains pixel data within the motion search window and determines the reference pixels used for prediction of the current coded block based on the motion search window. The positional relationship between the current frame, the reference frame, and the motion search window is as follows: Figure 2 As shown, GMV is the motion vector.

[0078] It should be noted that when establishing a motion search window centered on a reference block, if the reference block is a region close to the boundary of the reference frame, the determined motion search window may exceed the boundary of the reference frame. In this case, the size of the motion search window needs to be adjusted, for example, by adjusting the size of the motion search window according to the size of the reference frame, so that the determined reference pixels are the actual pixels existing in the reference frame. A consistent adjustment rule is used on both the encoding and decoding sides.

[0079] Through the above process, the reference search of the current coding block is no longer directed towards the entire reference frame, but rather towards a local region guided by geometric parameters. Based on the above analysis, the reference block constitutes the local search center, and the motion search window limits the range of reference pixels involved in the processing. This concentrates the cross-view search near the true parallax, maintaining matching compatibility under scene changes while controlling the number of candidate pixels and the size of the buffer.

[0080] Step 104: Determine the encoding result of the current encoding block based on the reference pixels.

[0081] The encoding result refers to the set of information that enables the decoder to reconstruct the current coding block. It includes at least one or more of the following: the prediction mode identifier used by the current coding block, reference block position information, residual information obtained after matching reference pixels, and entropy-coded data related to the residual representation. The prediction mode identifier characterizes the coding mode used when encoding the coding block, such as an inter-frame prediction mode. After determining the reference pixel in step 103, the encoder uses the prediction block formed by the reference pixel as the prediction value of the current coding block and subtracts the original pixel values ​​of the current coding block from the pixel values ​​of the prediction block sample by sample to obtain the residual block. If the current coding block contains both luma and chroma components, corresponding residual data are generated respectively; if the system uses a chroma prediction method with shared motion information, the chroma component uses the reference position of the luma component.

[0082] Subsequently, the encoder performs transform and quantization processing on the residual block. The transform can be an integer discrete cosine transform, a discrete sine transform, or a transform kernel allowed by the coding standard. Quantization uses a scalar quantization rule determined by the quantization parameter (QP) to obtain quantized residual coefficients. For the displacement description of the current coding block, the encoder can encode the information related to the final reference position. Then, the encoder inputs the prediction mode, block partitioning information, reference frame index, reference position information, and quantized residual coefficients into the entropy coding module, which uses CABAC (Context-based Adaptive Binary Arithmetic Coding), CAVLC (Context-Adaptive Variable Length Coding), or other binary entropy coding methods to generate a bitstream segment as part of the encoding result of the current coding block.

[0083] To ensure decoding consistency, the encoder can also perform inverse quantization and inverse transform on the quantization coefficients in the local reconstruction loop, add the reconstruction residual to the prediction block to obtain the reconstruction block, and write the reconstruction block to the reference frame buffer for subsequent processing of the current coding block or subsequent video frames. In this application, the coding result in technical terms is not a single numerical value, but rather a combination of the complete syntax and data used to recover the current coding block; the reference pixel is not a static image fragment, but actual reference information involved in generating the prediction block and residual. Based on the above analysis, it can be seen that the coding result of the current coding block is generated based on the reference pixels obtained by local window search, the prediction error revolves around the true disparity position after geometric constraints, and the bitstream organization remains consistent with the cross-viewpoint reference relationship.

[0084] In one possible implementation, a method for decoding multi-view video is also provided, such as... Figure 3 As shown, it includes:

[0085] Step 301: Receive the encoding results of the first video and the second video, as well as the motion vector between the second video and the first video. The video frames in the second video have corresponding reference frames in the first video.

[0086] Step 302: For the current decoding block of any video frame in the second video, determine the corresponding reference block in the reference frame corresponding to the targeted video frame according to the motion vector, set the motion search window with the reference block as the center, and determine the reference pixel;

[0087] Step 303: Based on the decoding result of the reference pixel, decode the encoding result of the current decoding block to obtain the decoding result of the current decoding block.

[0088] In this application, the decoding method is applied to a decoder, which can be integrated into a display terminal, a local player, a video decoding unit in a mobile SoC (System on Chip), or a remote receiving device. The decoder receives the bitstream from a transmission link or storage medium and parses the bitstream structure according to the transmission protocol: if the motion vector is encapsulated along with the encoding result, the encoding result of the first video stream, the encoding result of the second video stream, and the motion vector between them are extracted simultaneously from the bitstream; if the motion vector uses an independent transmission channel, the motion vector data needs to be obtained from the corresponding signaling or auxiliary stream. Subsequently, the decoder parses the frame-level syntax information related to the first and second video streams, recovering the frame order, reference relationships, and block-level coding mode, providing necessary parameters for subsequent reference pixel localization and block decoding based on the motion search window. Throughout the decoding process, the decoder determines the reference block in the reference frame based on the motion vector and dynamically sets the search window around the reference block, ultimately using the encoding result of the reference pixel to complete the reconstruction of the current decoded block.

[0089] For the current coded block in the second video stream, the decoder first locates the corresponding reference block in the corresponding reference frame of the first video stream based on the acquired motion vectors. Before decoding the current coded block of the second video stream, the decoder first decodes the corresponding reference frame based on the first video stream bitstream and stores the reconstructed pixel data of the reference frame in the reference frame buffer, thereby obtaining the pixel domain values ​​of the reference block. Subsequently, the decoder reads the encoding result of the current coded block, including the prediction mode identifier, reference position related information, quantization residual coefficients, and related entropy coding syntax. According to the geometric parameter configuration and syntax interpretation rules consistent with the encoder, the decoder determines the reference pixel extraction window (i.e., the "motion search window" in step 302, which is the area range set by the decoder to extract reference pixels and does not involve the matching search operation of the encoder) around the reference block according to preset rules to determine the reference pixels used to generate the prediction block.

[0090] Then, the decoder performs inverse entropy decoding, inverse quantization, and inverse transform on the quantized residual coefficients to obtain the reconstructed residual block. This reconstructed residual block is then sample-by-sample superimposed onto the prediction block to form the reconstruction result of the current decoding block. This reconstruction result is the decoding result of the current decoding block in step 303, and is written into the reconstruction buffer of the current frame of the second video stream for subsequent inter-block dependency processing or display output.

[0091] During the determination of the reference pixel region, if the reference range obtained after offsetting based on the motion vector exceeds the boundary of the reference frame, the decoder adopts the same boundary adjustment strategy as the encoder (e.g., cropping according to the reference frame size or performing boundary padding) to ensure that all pixels used are valid real pixels within the frame. If the bitstream contains implicit semantics that the window center originates from the geometric motion vector, the decoder directly reproduces the center position through preset or negotiated geometric parameters of the acquisition device, without needing to search again within the entire frame. Thus, the computational overhead of the decoder is mainly concentrated on syntax parsing, reference block reconstruction, and residual recovery, without involving the global candidate matching process. Based on the above processing, this application utilizes the reference block reconstructed from the first video stream and the encoding result of the current decoding block to collaboratively recover the content of the second video stream on the decoding side, enabling the cross-viewpoint local prediction relationship established at the encoder to be reversibly reproduced at the decoding side, thereby obtaining a decoding result that is completely consistent with the local reconstruction by the encoder.

[0092] Based on the above analysis, this application provides a multi-view video encoding and decoding method, including acquiring a first video and a second video synchronously acquired by an acquisition device, and establishing a temporal correspondence between video frames in the second video and corresponding reference frames in the first video; determining the motion vector between the second video and the first video based on the geometric parameters of the acquisition device; for the current encoding block of any video frame in the second video, determining a reference block in the corresponding reference frame based on the motion vector, and setting a motion search window centered on the reference block to determine reference pixels; determining the encoding result of the current encoding block based on the reference pixels; and on the decoder side, decoding the current encoding block based on the encoding result of the corresponding reference block in the reference frame and the encoding result of the current encoding block to obtain the decoding result. In this application, by incorporating the geometric parameters of the acquisition device into the process of determining the cross-view prediction starting point, and limiting the search area to a local window around the reference block, the reference read range on the encoder side, the size of the on-chip SRAM (Static Random Access Memory) cache, and the amount of DRAM (Dynamic Random Access Memory) access are restricted to an area more consistent with the actual parallax. The computational and storage overhead caused by global search in binocular scenes is controlled. At the same time, local search still retains the ability to correct for scene changes and installation deviations, so that the prediction of the current coded block is based on the combination of geometric guidance and local processing.

[0093] Based on the foregoing embodiments, further, such as Figure 4 As shown, based on the geometric parameters of the acquisition device, the motion vector between the second video stream and the first video stream is determined, including:

[0094] Step 401: Obtain the focal length, image distance, baseline distance, sensor width, and video resolution width of the acquisition device;

[0095] Step 402: Determine the resolution focal length based on the focal length, sensor width, and video resolution width;

[0096] Step 403: Determine the object distance between the imaging object and the acquisition device based on the focal length and image distance;

[0097] Step 404: Determine the motion vector based on the resolution focal length, object distance, and baseline distance.

[0098] In practice, the acquisition device can pre-store hardware parameters such as focal length, baseline distance and sensor width. The image distance can be output by the lens module at the current focus position or read in real time by the focus control module.

[0099] The resolution focal length can be calculated based on the ratio between the focal length and the sensor width, combined with the video resolution width, so that the same optical imaging relationship can be adapted to the current resolution.

[0100] The object distance can be determined based on the imaging relationship between the focal length and the image distance, and is updated as the zoom or focus state changes.

[0101] Furthermore, the motion vector can be determined by the resolution focal length, object distance, and baseline distance, and its value corresponds to the parallax displacement between the second video and the first video, and is used to indicate the offset position of the reference block in the reference frame.

[0102] This method first establishes a spatial mapping relationship between binocular viewpoints using geometric parameters during encoding, and then converts it into motion vectors for input into the reference block positioning process. This ensures that the search starting point is consistent with the actual parallax region, reduces invalid matching ranges, and keeps the encoding results consistent with the current shooting geometry.

[0103] In one specific embodiment, the acquisition device is a binocular camera of a VR headset. The encoder reads the focal length f, image distance v, baseline distance d, sensor width ω, and video resolution width W of the binocular camera, and calculates the object distance u based on the camera parameters. The definitions and relationships of the focal length f, image distance v, and object distance u of the binocular camera are as follows: Figure 5 As shown, F1 is the object-side focal point, and F is the image-side focal point. The binocular camera obtains the focal length f and image distance v of the current scene through the sensor. For example, when the camera focal length f is 26mm and the image distance v is 26.7mm, the encoder approximately calculates the object distance u ≈ 1000mm between the imaging object and the acquisition device according to the geometric optics formula 1 / u = 1 / f - 1 / v, and uses this object distance value as the basic parameter for subsequent global motion vector GMV calculation.

[0104] The encoder also calculates the resolution focal length Fp = f × W / ω based on the focal length f, sensor width ω, and video resolution width W. Finally, combining the object distance u and the distance between the two cameras d (i.e., the baseline distance), the motion vector of the second video channel relative to the first video channel is dynamically calculated using the formula GMV = Fp × d / u. This GMV value represents the overall translational parallax between the right and left video channels. In a practical application, the video resolution width W is typically 1920; the sensor uses a 1 / 3-inch format with a pixel size of 3.75μm, so the sensor width ω is 1920 × 3.75 = 7.2mm. If the currently calibrated optical focal length is the equivalent focal length of 26mm based on a 35mm film full-frame camera (reference width is 35mm), then the crop factor is 35 / 7.2 ≈ 4.86, and the corresponding actual physical focal length is 26 / 4.86 ≈ 5.35mm. Therefore, based on the resolution-focal-length conversion formula, we can calculate: Fp = 5.35 × 1920 ÷ 7.2 ≈ 1426 pixels. Finally, combining the above object distance u ≈ 1000mm and the distance d between the two cameras, i.e., the baseline distance (e.g., d = 60mm), we can obtain GMV ≈ 85.

[0105] Since the motion vector is determined by the focal length, image distance, baseline distance, sensor width, and video resolution width, the system can synchronously correct the reference block position when the focal length changes, the shooting is close, or the resolution changes, thereby reducing the amount of reference pixel readout and external storage access, and improving the matching accuracy and compression efficiency of binocular video predictive coding.

[0106] Based on the above method of determining motion vectors, the motion vectors can be redefined when changes in the geometric parameters of the acquisition device are detected or after a period of time.

[0107] In one possible implementation, the motion vector is determined based on the resolution focal length, object distance, and baseline distance, including:

[0108] Monitor the dynamic changes in the scene and adjust the weighting coefficients of resolution focal length, object distance, and baseline distance based on the monitoring results;

[0109] Based on the adjusted weighting coefficients, the motion vector is determined according to the resolution focal length, object distance, and baseline distance.

[0110] In practical implementation, the encoder continuously acquires the brightness distribution, target edge displacement, parallax change, or zoom status of the acquisition device for the current video frame, and monitors dynamic changes in the scene accordingly. When a close-range target rapidly enters the frame, the lens focal length changes, or parallax fluctuations intensify, the weight coefficients related to object distance are increased, or the weight coefficients corresponding to resolution focal length and baseline distance are increased simultaneously, making subsequent calculations more consistent with the current geometric relationships. When the scene changes smoothly, the weight coefficients remain at preset initial values. The weight coefficients can be read from a preset weight table in memory, or calculated based on dynamic changes through linear mapping, piecewise mapping, or normalization. The updated weights are then written into the calculation register of the current frame.

[0111] After weighting adjustments, the motion vector can be calculated using a weighted geometric relationship. This involves multiplying the resolution focal length, object distance, and baseline distance by their respective weighting coefficients and then combining the results to obtain the motion vector representing the offset of the second video stream relative to the first video stream. This motion vector serves as the starting point for the reference block search and is used to define the center position of the motion search window. Because the weighting coefficients can be corrected in real time according to scene changes, the calculated motion vector remains consistent with the actual parallax even when the target distance, lens parameters, or binocular baseline correspondence changes.

[0112] With this approach, motion vectors no longer rely on static geometric parameters but are dynamically updated as the scene changes. This makes the reference block positioning more accurate, the motion search window fits the real parallax region better, thereby reducing redundant access to reference pixels and improving the matching efficiency of binocular video coding.

[0113] It should be noted that when updating motion vectors, in addition to adjusting the weight coefficients based on the dynamic changes of the scene, the changes in the geometric parameters of the acquisition device can also be taken into account to determine the new motion vectors.

[0114] In one possible implementation, the method also includes:

[0115] Based on the monitoring results, the size of the motion search window is adjusted, and the reference pixel is determined according to the adjusted motion search window.

[0116] The monitoring results can be output by the scene monitoring module inside the encoder. This module generates monitoring values ​​based on continuous inter-frame differences, block-level motion consistency, or reference pixel offsets, and inputs these values ​​to the window adjustment unit. The window adjustment unit maps the monitoring values ​​to different window size parameters according to a preset threshold table; for example, it uses a larger search radius in regions with high dynamics and a smaller search radius in regions with low dynamics. The determination of the reference pixel can be completed by the pixel selection unit. This unit sequentially or in parallel reads candidate pixels within the adjusted motion search window and outputs the reference pixel closest to the current coding block. The selection result is further used to generate prediction information for the current coding block.

[0117] By adjusting the size of the motion search window based on monitoring results, the window range can adaptively change with scene variations, and reference pixels are determined within the adjusted local area, ensuring the search range remains consistent with actual matching requirements. This maintains the matching accuracy of reference pixels even when the scene is dynamically changing, reduces the reading of invalid candidate pixels, and thus lowers reference frame access overhead and encoding computation burden.

[0118] Based on the foregoing embodiments, further, such as Figure 6 As shown, based on the motion vector, a corresponding reference block is determined in the reference frame corresponding to the targeted video frame. A motion search window is set with the reference block as the center, and reference pixels are determined, including:

[0119] Step 601: Based on the horizontal and vertical components of the motion vector, determine the region in the reference frame that corresponds to the offset of the current coding block position, and define this region as the reference block;

[0120] Step 602: Use the geometric center of the reference block as the search starting point; according to the preset search range threshold, delineate a region centered on the search starting point in the reference frame as the motion search window;

[0121] Step 603: Perform pixel matching within the motion search window, and determine the pixel in the pixel block with the lowest matching cost as the reference pixel corresponding to the current coding block.

[0122] In the specific implementation, after the encoder receives the current coded block and its corresponding motion vector, it first maps the horizontal component to a horizontal offset in the reference frame and the vertical component to a vertical offset. Based on this, it determines a corresponding region within the reference frame that is the same size as or similar to the current coded block; this region serves as the reference block. Then, using the center point of the reference block as the search starting point, a rectangular or square motion search window is formed around this center point according to a preset search range threshold. When calculating the cost of candidate pixel blocks within the motion search window, absolute error, squared error, or pixel difference cost functions can be used. Finally, the pixel block with the minimum total cost is output as the reference pixel.

[0123] In this structure, the reference block is first located directly by motion vectors, and its center is used as a local search reference. The motion search window only covers a limited range of candidate regions, and the reference pixel is determined by the minimum cost matching result within the motion search window. This method gives the reference position determination and pixel matching clear spatial constraints, reduces unnecessary reference region access, and makes the matching result between the current coding block and the reference frame more stable.

[0124] Based on the foregoing embodiments, the preset search range threshold is further dynamically set according to the confidence level of the motion vector and / or the size of the current coded block:

[0125] When the confidence level of the motion vector is less than the first threshold, a first search range is set;

[0126] When the confidence level of the motion vector is greater than or equal to the first threshold, a second search range greater than the first search range is set.

[0127] The search range threshold is positively correlated with the size of the current coded block.

[0128] During operation, the encoder first determines the initial search range based on the motion vector and its confidence level corresponding to the current coded block, and then adjusts the threshold according to the current coded block size. A motion search window is then generated using the adjusted threshold, and matching is performed on reference pixels within this window. When the current coded block is small, the window shrinks to reduce the number of reference pixels read; when the current coded block is large or the motion vector confidence level is low, the window expands to accommodate larger offset candidates, thereby achieving adaptive coverage of the area near the reference block.

[0129] By adopting this method, the motion search window can be adjusted synchronously with the reliability of the motion vector and the current coding block size, so that the reference pixel search is consistent with the actual parallax range, reducing over-search or under-search, and making the reference pixel access range in binocular video coding more stable.

[0130] In one possible implementation, the encoder is integrated into an encoding chip, which also includes an on-chip buffer; after setting the motion search window centered on the reference block, and before determining the reference pixel, the following is also included:

[0131] The data read address is generated based on the pixel range covered by the motion search window in the reference frame;

[0132] Based on the data read address, pixel data within a pixel range is read from the external memory of the encoding chip and loaded into the on-chip cache;

[0133] When performing pixel matching within the motion search window, pixel data is read from the on-chip cache.

[0134] The encoding chip carries the encoder and its related processing flow, and the on-chip cache and external memory are configured together in the same encoding hardware. The on-chip cache is used to temporarily store pixel data read from the external memory. The data read address is used to indicate the access position of pixel data in the reference frame, which corresponds to the coverage area of ​​the motion search window in the reference frame, so as to perform directional data retrieval of the area to be matched within the motion search window. The external memory can be an independent storage unit outside the encoding chip, which stores the reference frame data. The on-chip cache can be a static random access memory unit or an equivalent cache unit inside the chip. The two work together to complete the handling and reuse of pixel data. In practical applications, other models of this component can also be selected, and this application does not limit them.

[0135] After the encoder completes reference block localization for the current coding block and establishes a motion search window centered on the reference block, the encoding chip generates continuous or discrete data read addresses based on the coordinate range covered by the motion search window in the reference frame, and reads the corresponding pixel data from external memory according to these addresses. The read data can be written to the on-chip cache row-by-row, block-by-block, or at a preset cache granularity, ensuring that pixels within the motion search window are pre-loaded before entering the matching stage. Subsequently, when performing pixel matching within the motion search window, the matching unit directly retrieves pixel data from the on-chip cache for cost calculation and reference pixel determination, without initiating real-time access to external memory.

[0136] This architecture pre-loads the pixels covering the motion search window into the on-chip buffer and only reads the buffer data during matching, thus centralizing the reference pixel determination process within the chip. This reduces the number of repeated read / write operations to external memory, lowers the reliance on off-chip bandwidth, and reduces dependence on large on-chip storage resources. Because pixel data is reused in the buffer, the encoding chip's storage organization can be more compact, thereby reducing chip area and improving hardware execution efficiency during stereo video encoding.

[0137] In one possible implementation, when the reference frame includes a forward reference frame and a backward reference frame:

[0138] Based on the motion vector, determine the first sub-reference block in the forward reference frame and set the first sub-search window; determine the second sub-reference block in the backward reference frame and set the second sub-search window.

[0139] Within the motion search window, pixel matching is performed, and the pixel block with the lowest matching cost is determined as the reference pixel for the current coding block, including:

[0140] Matching is performed in the first sub-search window and the second sub-search window respectively to obtain the first sub-pixel block and the second sub-pixel block;

[0141] The pixel with the smaller matching cost between the first and second sub-pixel blocks is selected as the final reference pixel; or the pixels in the first and second sub-pixel blocks are averaged to obtain the reference pixel.

[0142] The forward reference frame is used to provide candidate reference regions in the forward temporal direction relative to the current coding block, and the backward reference frame is used to provide candidate reference regions in the backward temporal direction relative to the current coding block. The first sub-reference block and the second sub-reference block are local reference regions located in the two types of reference frames based on motion vectors, respectively. The first sub-search window and the second sub-search window respectively limit the local matching range within the corresponding reference frame. The first sub-pixel block and the second sub-pixel block respectively represent the candidate matching results obtained within their respective sub-search windows.

[0143] In practical implementation, the encoder calculates the offset coordinate position in the forward reference frame based on the block address, size, and motion vector of the current coding block, and constructs a first sub-reference block using the pixel region corresponding to that position. Simultaneously, a first sub-search window is generated around this region within a preset boundary. Similarly, the encoder generates a second sub-reference block and a second sub-search window in the backward reference frame. Subsequently, the encoder performs pixel cost calculations within the two sub-search windows, comparing candidate pixel blocks based on the sum of absolute errors, mean square error, or other preset matching cost functions to obtain the first and second sub-pixel blocks. If a selection method is used, the matching costs of the two are compared, and the one with the smaller cost is selected as the final reference pixel. If a fusion method is used, pixel-by-pixel averaging or weighted averaging is performed on the corresponding pixels to generate reference pixels for prediction construction of the current coding block. The search window size, cost function form, and average weight used in this application can be set according to the coding configuration. In practical applications, these parameters can also be selected with other values, which are not limited in this application.

[0144] By establishing local reference blocks and sub-search windows in the forward and backward reference frames respectively, and optimizing or fusing the matching results on both sides, the selection of reference pixels can take into account bidirectional reference information, thereby improving the prediction consistency and matching accuracy in bidirectional reference scenarios, and reducing the dependence on large-scale search, thus reducing the amount of reference data access and encoding computation.

[0145] The following explanation uses an encoder deployed in an AR device as an example. In an AR device, the encoding chip is connected to a binocular camera, external storage, and a display module. The binocular camera simultaneously acquires a first video and a second video. The first video can be the left video, and the second video can be the right video. The video frames in the second video are associated with reference frames in the first video that correspond to the time or viewpoint. When a user wears the AR device to play augmented reality games, the device acquires stereoscopic video of the user's surrounding environment in real time, compresses the left and right video streams, and transmits them to the display module. The scene presents challenges such as transmitting binocular video with limited bandwidth, reducing bitrate, decreasing inter-frame prediction storage bandwidth and computational complexity, adapting to frequent user focus adjustments (e.g., observing nearby objects or parallax changes caused by movement), and ensuring compatibility with both ordinary displays and AR devices. During gameplay, users frequently adjust their head posture, such as turning left and right to observe the scene, or moving closer to or away from objects (e.g., picking up virtual items), resulting in dynamic changes in the binocular video parallax.

[0146] The encoding chip is used to process the acquired binocular video. The encoder reads the focal length f, image distance v, baseline distance d, sensor width ω, and video resolution width W from the acquisition device to determine the motion vector GMV between the second and first video streams. For example, when the resolution focal length Fp ≈ 1426 pixels, d = 35mm, and u ≈ 1000mm, the GMV ≈ 85. This GMV value indicates that the overall translational parallax between the right and left video streams is approximately 85 pixels.

[0147] In one possible implementation, the encoder continuously monitors dynamic changes in the scene, adjusts the weighting coefficients corresponding to the resolution focal length, object distance, and baseline distance based on the monitoring results, and then recalculates the motion vector using the adjusted weighting coefficients, so that the motion vector updates with scene changes. In a corresponding AR device example, when a user observes a nearby object, they frequently adjust the focal length. When the user turns their head left or right, or moves closer to or away from the object, it causes dynamic changes in parallax. The encoder updates the GMV in real time accordingly to adapt to the parallax changes in the dynamic scene.

[0148] For any video frame in the second video stream, the encoder reads the horizontal and vertical components of the motion vector in the current encoded block. It then determines the region in the corresponding reference frame that has shifted relative to the current encoded block position and designates this region as the reference block. The geometric center of the reference block is used as the search starting point, and a motion search window is defined in the reference frame centered on this starting point. For the AR device example, a reduced motion search window is set centered on the reference block position pointed to by the GMV. For instance, when the GMV points to the (100, 100) position in the left-side image, a search range of ±4 pixels is defined centered on this point; traditional solutions typically use ±64 pixels. The search range threshold is dynamically set based on the confidence level of the motion vector or the size of the current encoded block. When the motion vector value is less than the first threshold, a first search range is used; when the motion vector value is greater than or equal to the first threshold, a second search range larger than the first search range is used. Furthermore, the larger the current encoded block, the larger the search range threshold. The encoder also adjusts the size of the motion search window in sync with the scene dynamic monitoring results, and performs pixel matching within the adjusted motion search window, determining the pixel block with the lowest matching cost as the reference pixel corresponding to the current encoding block.

[0149] When the encoder is integrated into the encoding chip and the encoding chip includes an on-chip buffer, after determining the motion search window, the encoder generates a data read address based on the pixel range covered by the motion search window in the reference frame. Then, based on the data read address, it reads the corresponding pixel data from the external memory of the encoding chip and loads it into the on-chip buffer. Subsequently, during pixel matching within the motion search window, pixel data is directly read from the on-chip buffer. The current block of the right-channel video matches the reference block of the left-channel video through a narrowed search window, generating residual data and compressing it. For example, if the current block of the right-channel video matches a reference block in the left-channel video, only the residual information of the pixel difference between the two is stored. The encoder performs inter-frame prediction, residual generation, transform, quantization, and entropy coding on the current encoding block based on the determined reference pixels to obtain the encoding result of the current encoding block. The decoder obtains the encoding result of the current encoding block and, in conjunction with the encoding result of the corresponding reference block in the reference frame, decodes the current encoding block to obtain the decoding result of the current encoding block.

[0150] When the reference frame includes both a forward reference frame and a backward reference frame, the encoder determines a first sub-reference block and sets a first sub-search window in the forward reference frame based on the same motion vector, and determines a second sub-reference block and sets a second sub-search window in the backward reference frame. Matching is then performed within both the first and second sub-search windows to obtain a first sub-pixel block and a second sub-pixel block. The encoder can select the sub-pixel block with the lower matching cost as the reference pixel, or it can average the pixels in the first and second sub-pixel blocks to obtain the reference pixel, and use this to complete the encoding of the current coding block. When decoding the current coding block, the decoder combines the encoding result of the corresponding reference block with the encoding result of the current coding block to recover the current coding block.

[0151] In temporal scalable coding, the left-channel video is encoded as the base layer, and the right-channel video is encoded as the enhancement layer. For example... Figure 7 As shown, L0-L8 represent the left-channel video capture, and R0-R8 represent the right-channel video capture. During encoding, frame L0 is used as the base layer for independent encoding, while R0 references L0. The motion vector GMV is simultaneously incorporated into the R0 encoding, utilizing the spatial translation correlation of the multi-view video for R0 encoding. L1, on the other hand, references L0 encoding and utilizes the temporal correlation of the video for compression. R1 references L1 frames, and the currently calculated GMV is transmitted during encoding. This process sequentially completes the encoding of the left and right-channel video captures, combining the two video streams into a single output video. The base and enhancement layers in the synthesized stream are decoded and output to the left and right display modules of the AR device, respectively. Specifically, frame L0 is first decoded, then frame R0 is decoded with reference to frame L0. After the decoding of frame R0 is complete, the decoding results of frame R0 and frame L0 are simultaneously sent to the right and left display modules, respectively, to ensure synchronized video playback. If the display device only supports single-viewpoint playback, the enhancement layer is discarded and only the base layer video is output; if the display device is a binocular display device such as an AR device, both the base layer and the enhancement layer are extracted and output separately, thus adapting between ordinary displays and AR devices.

[0152] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0153] Based on the same inventive concept, this application also provides a multi-view video encoding and decoding device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more multi-view video encoding and decoding device embodiments provided below can be found in the limitations of the encoding and decoding methods of multi-view video above, and will not be repeated here.

[0154] Figure 8 A schematic diagram of the structure of the multi-view video encoding and decoding device provided in this application is shown below. Figure 8 As shown, the multi-view video encoding and decoding device provided in this embodiment includes:

[0155] The data acquisition module 801 is used to acquire the first video and the second video synchronously acquired by the acquisition device. The video frames in the second video have corresponding reference frames in the first video.

[0156] Vector determination module 802 is used to determine the motion vector between the second video and the first video based on the geometric parameters of the acquisition device;

[0157] The search and matching module 803 is used to determine the corresponding reference block in the reference frame corresponding to the target video frame based on the motion vector for the current coding block of any video frame in the second video, and to set a motion search window centered on the reference block to determine the reference pixel.

[0158] The encoding module 804 is used to determine the encoding result of the current encoding block based on the reference pixels;

[0159] The decoding module 805 is used to decode the current coding block based on the encoding result of the corresponding reference block in the reference frame and the encoding result of the current coding block, so as to obtain the decoding result of the current coding block.

[0160] In one possible implementation, the vector determination module 802 is specifically used for:

[0161] Acquire the focal length, image distance, baseline distance, sensor width, and video resolution width of the acquisition device;

[0162] Determine the resolution focal length based on focal length, sensor width, and video resolution width;

[0163] Based on the focal length and image distance, determine the object distance between the imaging object and the acquisition device;

[0164] The motion vector is determined based on the resolution focal length, object distance, and baseline distance.

[0165] In one possible implementation, the vector determination module 802 is specifically used for:

[0166] Monitor the dynamic changes in the scene and adjust the weighting coefficients of resolution focal length, object distance, and baseline distance based on the monitoring results;

[0167] Based on the adjusted weighting coefficients, the motion vector is determined according to the resolution focal length, object distance, and baseline distance.

[0168] In one possible implementation, the search matching module 803 is specifically used for:

[0169] Based on the monitoring results, the size of the motion search window is adjusted, and the reference pixel is determined according to the adjusted motion search window.

[0170] In one possible implementation, the search matching module 803 is specifically used for:

[0171] Based on the horizontal and vertical components of the motion vector, the region in the reference frame corresponding to the offset of the current coded block position is determined, and this region is determined as the reference block;

[0172] The geometric center of the reference block is used as the starting point for the search;

[0173] Based on the preset search range threshold, a region centered on the search starting point is defined in the reference frame as the motion search window;

[0174] Pixel matching is performed within the motion search window, and the pixel in the pixel block with the lowest matching cost is determined as the reference pixel corresponding to the current coding block.

[0175] In one possible implementation, the search matching module 803 is specifically used for:

[0176] When the confidence level of the motion vector is less than the first threshold, a first search range is set;

[0177] When the confidence level of the motion vector is greater than or equal to the first threshold, a second search range greater than the first search range is set.

[0178] And / or, the search range threshold is positively correlated with the size of the current coded block.

[0179] In one possible implementation, the encoder is integrated into the encoding chip, which also includes an on-chip buffer; the search matching module 803 is further used for:

[0180] The data read address is generated based on the pixel range covered by the motion search window in the reference frame;

[0181] Based on the data read address, pixel data within a pixel range is read from the external memory of the encoding chip and loaded into the on-chip cache;

[0182] When performing pixel matching within the motion search window, pixel data is read from the on-chip cache.

[0183] In one possible implementation, the search matching module 803 is specifically used for:

[0184] When the reference frame includes a forward reference frame and a backward reference frame:

[0185] Based on the motion vector, determine the first sub-reference block in the forward reference frame and set the first sub-search window; determine the second sub-reference block in the backward reference frame and set the second sub-search window.

[0186] Within the motion search window, pixel matching is performed, and the pixels in the pixel block with the lowest matching cost are determined as the reference pixels for the current coding block, including:

[0187] Matching is performed in the first sub-search window and the second sub-search window respectively to obtain the first sub-pixel block and the second sub-pixel block;

[0188] The pixel with the smaller matching cost in the first and second sub-pixel blocks is selected as the final reference pixel; or the pixels in the first and second sub-pixel blocks are averaged to obtain the reference pixel.

[0189] The multi-view video encoding and decoding device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0190] This application also provides an encoder for performing the above-described multi-view video encoding method.

[0191] This application also provides a decoder for performing the above-described multi-view video decoding method.

Claims

1. A method for encoding multi-view video, characterized in that, Applied to an encoder, the method includes: Acquire the first and second video streams synchronously acquired by the acquisition device, wherein video frames in the second video stream have corresponding reference frames in the first video stream; Based on the geometric parameters of the acquisition device, determine the motion vector between the second video stream and the first video stream; For the current encoding block of any video frame in the second video, a corresponding reference block is determined in the reference frame corresponding to the targeted video frame based on the motion vector, and a motion search window is set with the reference block as the center to determine the reference pixel; The encoding result of the current encoding block is determined based on the reference pixel.

2. The method according to claim 1, characterized in that, Determining the motion vector between the second video stream and the first video stream based on the geometric parameters of the acquisition device includes: Obtain the focal length, image distance, baseline distance, sensor width, and video resolution width of the acquisition device; The resolution focal length is determined based on the focal length, sensor width, and video resolution width. Based on the focal length and the image distance, the object distance between the imaging object and the acquisition device is determined; The motion vector is determined based on the resolution focal length, the object distance, and the baseline distance.

3. The method according to claim 2, characterized in that, The determination of the motion vector based on the resolution focal length, the object distance, and the baseline distance includes: Monitor the dynamic changes in the scene, adjust the weighting coefficients corresponding to the resolution focal length, the object distance, and the baseline distance based on the monitoring results, and determine the motion vector based on the adjusted weighting coefficients, according to the resolution focal length, the object distance, and the baseline distance; and / or, Based on the monitoring results, the size of the motion search window is adjusted, and a reference pixel is determined according to the adjusted motion search window.

4. The method according to claim 1, characterized in that, The step of determining a corresponding reference block in the reference frame corresponding to the targeted video frame based on the motion vector, setting a motion search window centered on the reference block, and determining reference pixels includes: Based on the horizontal and vertical components of the motion vector, a region corresponding to the offset of the current coding block position is determined in the reference frame, and this region is determined as the reference block; The geometric center of the reference block is used as the starting point for the search; Based on a preset search range threshold, a region centered on the search starting point is defined in the reference frame as the motion search window; Within the motion search window, pixel matching is performed, and the pixel in the pixel block with the lowest matching cost is determined as the reference pixel corresponding to the current coding block.

5. The method according to claim 4, characterized in that, The preset search range threshold is dynamically set based on the confidence level of the motion vector and / or the size of the current coding block; When the confidence level of the motion vector is less than a first threshold, a first search range is set; When the confidence level of the motion vector is greater than or equal to the first threshold, a second search range greater than the first search range is set. The search range threshold is positively correlated with the size of the current coded block.

6. The method according to claim 4, characterized in that, The encoder is integrated into the encoding chip, which also includes an on-chip buffer; after setting the motion search window centered on the reference block and before determining the reference pixel, the method further includes: A data read address is generated based on the pixel range covered by the motion search window in the reference frame; Based on the data read address, pixel data within the pixel range is read from the external memory of the encoding chip and loaded into the on-chip cache; When performing pixel matching within the motion search window, pixel data is read from the on-chip cache.

7. The method according to claim 4, characterized in that, When the reference frame includes a forward reference frame and a backward reference frame: Based on the motion vector, a first sub-reference block is determined in the forward reference frame and a first sub-search window is set; a second sub-reference block is determined in the backward reference frame and a second sub-search window is set. The step of performing pixel matching within the motion search window and determining the pixel in the pixel block with the lowest matching cost as the reference pixel corresponding to the current coding block includes: Matching is performed within the first sub-search window and the second sub-search window respectively to obtain the first sub-pixel block and the second sub-pixel block; Pixels from the first and second sub-pixel blocks with the smaller matching cost are selected as the final reference pixels; or the pixels from the first and second sub-pixel blocks are averaged to obtain the reference pixels.

8. A method for decoding multi-view video, characterized in that, Applied to a decoder, the method includes: The encoding results of the first video and the second video, as well as the motion vector between the second video and the first video, are received. The video frames in the second video have corresponding reference frames in the first video. For the current decoding block of any video frame in the second video, a corresponding reference block is determined in the reference frame corresponding to the targeted video frame according to the motion vector, and a motion search window is set with the reference block as the center to determine the reference pixel; Based on the decoding result of the reference pixel, the encoding result of the current decoding block is decoded to obtain the decoding result of the current decoding block.

9. An encoder, characterized in that, Encoding method for performing multi-view video as described in any one of claims 1-7.

10. A decoder, characterized in that, Used to perform the multi-view video decoding method as described in claim 8.

Images