Video compression method, decompression method, device, electronic device and storage medium

Abstract

A video compression method, a decompression method, an apparatus, an electronic device and a storage medium, wherein the video compression method comprises: obtaining a free viewpoint video frame, wherein the video frame comprises a spliced ​​image formed by a texture map of multiple synchronized viewpoints and a depth map of the corresponding viewpoint; identifying a texture map area and a depth map area in the spliced ​​image; encoding the texture map area using a preset first encoding method, and encoding the depth map area using a preset second encoding method to obtain a compressed video frame; wherein the second encoding method is an encoding method based on a region of interest. The above scheme can reduce the compression loss of the image, thereby improving the image quality of the free viewpoint video.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of video processing technology, and in particular to a video compression method, a decompression method, a device, an electronic device, and a storage medium. Background Art

[0002] Video data is data that supports video playback for viewing by users. Generally, video data only supports viewing by users from one viewing angle, and users cannot adjust the viewing angle.

[0003] Free viewpoint video is a technology that can provide a high-freedom viewing experience. Users can adjust the viewing angle through interactive operations during the viewing process and watch from the free viewpoint they want, which can greatly improve the viewing experience.

[0004] At present, in order to realize the playback of free viewpoint video, there is a free viewpoint video generation method based on the splicing of texture map and depth map. Specifically, the texture maps of multiple synchronized perspectives and the depth maps of corresponding perspectives are spliced, the spliced ​​images of multiple frame moments are uniformly compressed and transmitted to the terminal for decoding, as well as virtual viewpoint reconstruction based on the depth map (Depth Image Based Rendering, DIBR).

[0005] However, the current encoding method for the free viewpoint video obtained by stitching the texture map and the depth map will result in a large compression loss in the stitched image, which will have a great impact on the image quality of the free viewpoint video reconstructed by the terminal. Summary of the invention

[0006] In view of this, the embodiments of the present specification provide a video compression method, a decompression method, an apparatus, a device and a storage medium, which can reduce the compression loss of an image and thus improve the image quality of a free viewpoint video.

[0007] The embodiment of this specification provides a video compression method, including:

[0008] Acquire a free viewpoint video frame, wherein the video frame includes a spliced ​​image formed by texture maps of multiple synchronized viewpoints and depth maps of corresponding viewpoints;

[0009] Identifying a texture map region and a depth map region in the stitched image;

[0010] The texture map area is encoded using a preset first encoding method, and the depth map area is encoded using a preset second encoding method to obtain a compressed video frame; wherein the second encoding method is an encoding method based on a region of interest.

[0011] Optionally, encoding the depth map area using a preset second encoding method includes:

[0012] The ROI pixel area in the depth map area is encoded based on a preset constant quantization parameter.

[0013] Optionally, encoding the depth map area using a preset second encoding method includes:

[0014] A first constant quantization parameter is used to encode the foreground edge pixel area in the depth map sub-area corresponding to each viewpoint contained in the depth map area, and a second constant quantization parameter is used to encode the non-foreground edge pixel area in the depth map sub-area corresponding to each viewpoint, and the parameter value of the first constant quantization parameter is less than the parameter value of the second constant quantization parameter.

[0015] Optionally, encoding the depth map area using a preset second encoding method includes:

[0016] For the ROI pixel area in the depth map area, a preset constant quality factor is used for encoding.

[0017] Optionally, the texture map area includes a special effect rendering pixel area; and encoding the texture map area using a preset first encoding method includes:

[0018] The special effect rendering pixel area in the texture map area is encoded using an encoding method suitable for augmented reality special effects.

[0019] Optionally, the method further includes: using a constant bit rate for the stitched image.

[0020] Optionally, the pixel points of the depth map sub-region in the stitched image are all or part of the pixel points in a set of pixel points that correspond one-to-one to the pixel points in the texture map sub-region of the corresponding viewpoint.

[0021] Optionally, the texture map area includes a plurality of texture map sub-areas corresponding to viewpoints, the depth map area includes a plurality of depth map sub-areas corresponding to viewpoints, and the depth map sub-areas are smaller than the texture map sub-areas.

[0022] Optionally, the depth map sub-region in the stitched image is obtained by downsampling an original depth map that corresponds one-to-one to pixel points in the texture map of the corresponding viewpoint.

[0023] The embodiment of this specification also provides a video decompression method, including:

[0024] Obtaining a compressed video frame and a quantization parameter of a pixel block in the compressed video frame, wherein the compressed video frame includes a texture map area encoded by a first encoding method and a depth map area encoded by a second encoding method, wherein the texture map area includes texture maps of multiple synchronized viewpoints, the depth map area includes depth maps of viewpoints corresponding to each texture map, and the second encoding method is an encoding method based on a region of interest;

[0025] Based on the quantization parameters of the pixel blocks in the compressed video frame, the compressed video frame is decompressed according to the pixel blocks to obtain a spliced ​​image of the free viewpoint video frame, wherein the spliced ​​image includes the texture map areas of the synchronized multiple viewpoints and the depth map areas of the corresponding viewpoints.

[0026] The embodiment of this specification also provides a video compression device, including:

[0027] A video frame acquisition unit, adapted to acquire a free viewpoint video frame, wherein the video frame includes a spliced ​​image formed by texture maps of multiple viewpoints and depth maps of corresponding viewpoints;

[0028] an identification unit, adapted to identify a texture map region and a depth map region in the stitched image;

[0029] A first encoding unit, encoding the texture map region by using a preset first encoding method;

[0030] The second encoding unit encodes the depth map area by adopting a preset second encoding method, where the second encoding method is an encoding method based on a region of interest.

[0031] The embodiment of this specification also provides a video decompression device, including:

[0032] an acquisition unit, adapted to acquire a compressed video frame and a quantization parameter of a pixel block in the compressed video frame, wherein the compressed video frame includes a texture map region encoded by a first encoding method and a depth map region encoded by a second encoding method, wherein the texture map region includes texture maps of multiple synchronized viewpoints, the depth map region includes depth maps of viewpoints corresponding to each texture map, and the second encoding method is an encoding method based on a region of interest;

[0033] A decompression unit is adapted to decompress the compressed video frame according to the pixel blocks based on the quantization parameters of the pixel blocks in the compressed video frame to obtain a spliced ​​image of the free viewpoint video frame, wherein the spliced ​​image includes the texture map areas of the synchronized multiple viewpoints and the depth map areas of the corresponding viewpoints.

[0034] An embodiment of the present specification also provides an electronic device, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, wherein the processor executes the steps of the method described in any of the above embodiments when executing the computer instructions.

[0035] The embodiments of the present specification also provide a computer-readable storage medium on which computer instructions are stored, wherein the computer instructions, when executed, execute the steps of the method described in any of the aforementioned embodiments.

[0036] Compared with the prior art, the technical solution of the embodiment of this specification has the following beneficial effects:

[0037] Using the video compression method of the embodiment of the present specification, different encoding methods are used to compress the texture map area and the depth map area in the stitched image of the video frame, wherein the depth map area is compressed using an encoding method based on the region of interest, which can be compressed according to the image features of the depth map area, thereby reducing the compression loss of the depth map, and further improving the image quality of the free viewpoint video reconstructed based on the depth map.

[0038] Furthermore, encoding the ROI pixel area in the depth map area based on a preset constant quantization parameter can reduce the compression loss of the ROI pixel area. On the basis of compressing the depth map area as much as possible, the compression loss of the depth map area of ​​the stitched image can be reduced.

[0039] Furthermore, a first constant quantization parameter is used to encode the foreground edge pixel area in the depth map sub-region corresponding to each viewpoint contained in the depth map region, and a second constant quantization parameter is used to encode the non-foreground edge pixel area in the depth map sub-region corresponding to each viewpoint. Since the foreground edge pixel area in the depth map sub-region is very critical to the reconstruction quality of the free viewpoint video, the parameter value of the first constant quantization parameter used for the foreground edge pixel area in the depth map sub-region is smaller than the parameter value of the second quantization parameter used for the non-foreground edge pixel area in the depth map sub-region, which can reduce the compression loss of the depth map region and thereby improve the reconstruction quality of the reconstructed free viewpoint video. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of a specific application system for free viewpoint video display in an embodiment of this specification;

[0041] Figure 2 This is a schematic diagram of an interactive interface of a terminal device in an embodiment of this specification;

[0042] Figure 3It is a schematic diagram of a collection device setting method in an embodiment of this specification;

[0043] Figure 4 This is another schematic diagram of the interactive interface of a terminal device in an embodiment of this specification;

[0044] Figure 5 is a schematic diagram of a free viewpoint video data generation process in an embodiment of this specification;

[0045] Figure 6 is a schematic diagram of generation and processing of 6DoF video data in an embodiment of this specification;

[0046] Figure 7 It is a structural diagram of a data header file in an embodiment of this specification;

[0047] Figure 8 is a schematic diagram of 6DoF video data processing on the user side in an embodiment of this specification;

[0048] Fig. 9 is a structural schematic diagram of a spliced ​​image of a video frame in an embodiment of this specification;

[0049] Fig.10 is a flow chart of a video compression method in an embodiment of this specification;

[0050] Fig.11 is a structural schematic diagram of another spliced ​​image in an embodiment of this specification;

[0051] Fig.12 is a structural schematic diagram of another spliced ​​image in an embodiment of this specification;

[0052] Fig.13 is a structural schematic diagram of another spliced ​​image in an embodiment of this specification;

[0053] Fig.14 is a structural schematic diagram of another spliced ​​image in an embodiment of this specification;

[0054] Fig.15 is a structural schematic diagram of another spliced ​​image in an embodiment of this specification;

[0055] Fig.16 is a schematic diagram of pixel data distribution of a texture map in an embodiment of this specification;

[0056] Fig.17 is a schematic diagram of pixel data distribution of another texture map in an embodiment of this specification;

[0057] Fig.18 is a schematic diagram of data storage in a spliced ​​image in an embodiment of the present invention;

[0058] Fig.19 is a schematic diagram of another data storage in a spliced ​​image according to an embodiment of the present invention;

[0059] Fig. 20 is a schematic diagram of a video compression method in a specific application scenario in an embodiment of this specification;

[0060] Fig.21 is a flow chart of a video decompression method in an embodiment of this specification;

[0061] Fig. 22 is a structural schematic diagram of a video compression device in an embodiment of this specification;

[0062] Fig.23 It is a structural schematic diagram of a video decompression device in an embodiment of this specification;

[0063] Fig.24 It is a structural schematic diagram of an electronic device in an embodiment of this specification;

[0064] Fig.25 It is a structural diagram of a video processing system in an embodiment of this specification. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand and implement the embodiments in this specification, the following first provides an exemplary introduction to the implementation of free viewpoint video in conjunction with the accompanying drawings and specific application scenarios.

[0066] refer to Figure 1 In the embodiment of the present invention, a specific application system for free viewpoint video display may include a collection system 11, a server 12 and a display device 13 of multiple collection devices, wherein the collection system 11 may collect images of the viewing area; the collection system 11 or the server 12 may process the acquired synchronized multiple texture maps to generate multi-angle free viewpoint data that can support the display device 13 to switch virtual viewpoints. The display device 13 may display a reconstructed image generated based on the multi-angle free viewpoint data, the reconstructed image corresponds to a virtual viewpoint, and may display reconstructed images corresponding to different virtual viewpoints according to user instructions, switching the viewing position and viewing angle.

[0067] In a specific implementation, the process of reconstructing an image and obtaining a reconstructed image can be implemented by the display device 13, or by a device located in a content delivery network (Content Delivery Network, CDN) in an edge computing manner. It is understandable that Figure 1 This is only an example and is not intended to limit the collection system, server, terminal device, or specific implementation method.

[0068] Continue to refer Figure 1 , the user can view the area to be viewed through the display device 13. In this embodiment, the area to be viewed is a basketball court. As mentioned above, the viewing position and viewing angle can be switched.

[0069] For example, the user can slide on the screen to switch the virtual viewpoint. Figure 2 , the user's finger moves along D 22 When you slide the screen in the right direction, you can switch the virtual viewpoint for viewing. Figure 3 , the position of the virtual viewpoint before sliding may be VP1, and after sliding the screen to switch the virtual viewpoint, the position of the virtual viewpoint may be VP2. Figure 4 After sliding the screen, the reconstructed image displayed on the screen can be Figure 4 The reconstructed image can be obtained by reconstructing the image based on multi-angle free view data generated by images collected by multiple collection devices in an actual collection scenario.

[0070] It is understandable that the image viewed before switching may also be a reconstructed image. The reconstructed image may be a frame image in a video stream. In addition, the method of switching the virtual viewpoint according to the user's instruction may be various, which is not limited here.

[0071] In a specific implementation, the viewpoint can be represented by 6 degrees of freedom (DoF) coordinates, where the spatial position of the viewpoint can be represented as (x, y, z) and the viewing angle can be represented as three rotation directions: Accordingly, based on the 6-DOF coordinates, the virtual viewpoint, including the position and viewing angle, can be determined.

[0072] Virtual viewpoint is a three-dimensional concept, and three-dimensional information is required to generate a reconstructed image. In a specific implementation, the multi-angle free viewpoint data may include depth map data to provide the third dimension information outside the plane image. Compared with other implementations, such as providing three-dimensional information through point cloud data, the amount of depth map data is relatively small.

[0073] In the embodiment of the present invention, the switching of the virtual viewpoint can be performed within a certain range, which is the multi-angle free viewing angle range. That is, within the multi-angle free viewing angle range, the position and viewing angle of the virtual viewpoint can be switched arbitrarily.

[0074] The multi-angle free viewing angle range is related to the arrangement of the acquisition equipment. The wider the shooting coverage of the acquisition equipment, the larger the multi-angle free viewing angle range. The image quality displayed by the terminal device is related to the number of acquisition devices. Generally, the more acquisition devices are set, the fewer the hollow areas in the displayed image.

[0075] In addition, the range of the multi-angle free viewing angle is related to the spatial distribution of the acquisition device. The range of the multi-angle free viewing angle and the interaction mode with the display device on the terminal side can be set based on the spatial distribution relationship of the acquisition device.

[0076] Those skilled in the art will appreciate that the above embodiments and corresponding drawings are for illustrative purposes only and are not limitations on the settings of acquisition devices and the relationship between multi-angle free viewing angles, nor are they limitations on the interaction methods and display effects of display devices.

[0077] Combined with reference Figure 5 In order to reconstruct free viewpoint video, texture image acquisition and depth map calculation are required, which includes three main steps: multi-camera video acquisition (Multi-camera Video Capturing), camera internal and external parameter calculation (Camera Parameter Estimation), and depth map calculation (Depth Map Calculation). For multi-camera video acquisition, it is required that the videos acquired by each camera can be aligned at the frame level. Among them, the texture image (Texture Image) can be obtained through multi-camera video acquisition; the camera parameters (Camera Parameter) can be obtained through camera internal and external parameter calculation. The camera parameters can include camera internal parameter data and external parameter data; the depth map (Depth Map) can be obtained through depth map calculation. The texture images of multiple synchronized perspectives and the depth maps and camera parameters of the corresponding perspectives form 6DoF video data.

[0078] In the embodiments of this specification, no special camera, such as a light field camera, is required to capture the video. Similarly, no complicated camera calibration is required before the capture. The positions of multiple cameras can be arranged to better capture the objects or scenes to be captured.

[0079] After the above three steps are completed, the texture map collected from multiple cameras, the camera parameters of all cameras, and the depth map of each camera are obtained. These three parts of data can be called data files in multi-angle free-view video data, or 6-degree-of-freedom video data (6DoF video data). With this data, the user end can generate virtual viewpoints based on the virtual 6-degree-of-freedom (DoF) position, thereby providing a 6DoF video experience.

[0080] Combined with reference Figure 6, 6DoF video data and indicative data can be compressed and transmitted to the user side, and the user side can obtain the user side 6DoF expression, that is, the aforementioned 6DoF video data and metadata, based on the received data. Among them, the indicative data can also be called metadata (Metadata), wherein the video data includes texture maps and depth map data of each viewpoint corresponding to multiple cameras, and the texture map and the depth map can be spliced ​​according to certain splicing rules or splicing patterns to form a spliced ​​image.

[0081] Combined with reference Figure 7 , metadata can be used to describe the data mode of 6DoF video data, which can include: stitching pattern metadata, which is used to indicate the storage rules of pixel data of multiple texture maps and depth map data in the stitched image; padding pattern metadata, which can be used to indicate the way to perform edge protection in the stitched image, and other metadata. Metadata can be stored in the data header file, and the specific storage order can be as follows: Figure 7 as shown, or stored in another order.

[0082] Combined with reference Figure 8 , the user side obtains 6DoF video data, which includes camera parameters, stitched images (texture maps and depth maps), and metadata (metadata), in addition to the user's interactive behavior data. With this data, the user side can use 6DoF rendering based on depth image rendering (DIBR, Depth Image-Based Rendering) to generate a virtual viewpoint image at a specific 6DoF position generated by the user's interactive behavior, that is, according to the user's instruction, determine the virtual viewpoint at the 6DoF position corresponding to the instruction.

[0083] Reference Fig. 9 A structural schematic diagram of a spliced ​​image of a video frame is shown. In some specific implementations, the texture maps and corresponding depth maps of multiple synchronized viewpoints are spliced, and the spliced ​​image is compressed using a unified encoding method and transmitted to a terminal side (such as a display device), where decoding and DIBR image reconstruction are performed.

[0084] The inventors have found through research that, due to the limitation of transmission bandwidth, using a unified encoding method to compress the spliced ​​images will cause compression loss of the depth map, which will have a great impact on the image quality of the virtual viewpoint reconstructed by DIBR.

[0085] In response to the above problems, in the solution adopted in the embodiments of the present specification, different encoding methods are used to compress the texture map area and the depth map area in the stitched image of the video frame, wherein the depth map area is compressed using an encoding method based on the region of interest (RoI), which can be compressed according to the image features of the depth map area, thereby reducing the compression loss of the depth map, and further improving the image quality of the free viewpoint video reconstructed based on the depth map.

[0086] Reference Fig.10 The flowchart of the video decompression method shown in FIG. 1 is a flowchart of the video decompression method. In a specific implementation, for a free viewpoint video frame, the following steps can be used to perform video compression:

[0087] S101, obtaining a free viewpoint video frame, wherein the video frame includes a spliced ​​image formed by texture maps of multiple synchronized viewpoints and depth maps of corresponding viewpoints.

[0088] In a specific implementation, the free viewpoint video frame may be formed by stitching a texture map and a depth map. As described in the previous embodiment of the multi-angle free viewpoint video, such as a 6DoF video frame, each video frame may include a stitched image formed by stitching a texture map of multiple viewpoints and a depth map of the corresponding viewpoint. In one embodiment of the present specification, the structure of the stitched image contained in the video frame is as follows: Fig. 9 As shown, it includes synchronized texture maps of 8 viewpoints and corresponding depth maps of 8 viewpoints.

[0089] S102: Identify a texture map area and a depth map area in the stitched image.

[0090] In a specific implementation, the texture map area and the depth map area in the stitched image can be identified by image feature recognition or by obtaining the stitching mode metadata of the stitched image. Fig. 9 As shown, it can be identified that the upper half area of ​​the spliced ​​image is the texture map area, and the lower half area is the depth map area.

[0091] S103, encoding the texture map area using a preset first encoding method, and encoding the depth map area using a preset second encoding method, to obtain a compressed video frame; wherein the second encoding method is an encoding method based on a region of interest.

[0092] In some embodiments of the present specification, the ROI pixel area in the depth map area may be encoded based on a preset constant quantization parameter (CQP).

[0093] As mentioned above, the stitched image is formed by stitching together the texture maps of multiple synchronized viewpoints and the depth maps of the corresponding viewpoints, wherein the texture map area corresponding to each viewpoint can be called the texture map sub-area of ​​the corresponding viewpoint, and the depth map area corresponding to the corresponding viewpoint can be called the depth map sub-area of ​​the corresponding viewpoint.

[0094] The inventor has found through research that the quality of the foreground edge pixel area in the depth map sub-area is very critical to the image quality reconstructed based on the virtual viewpoint. Based on this, in order to improve the image quality of the reconstructed virtual viewpoint, in some embodiments of the present specification, the foreground edge pixel area in the depth map sub-area corresponding to each viewpoint contained in the depth map area is encoded using a first constant quantization parameter, and the non-foreground edge pixel area in the depth map sub-area corresponding to each viewpoint is encoded using a second constant quantization parameter, and the parameter value of the first constant quantization parameter is less than the parameter value of the second constant quantization parameter.

[0095] Among them, the larger the quantization parameter (QP) value is, the larger the quantization step is, and the lower the image quality of the compressed video obtained after encoding is. A QP of 0 indicates lossless encoding. When CQP is used, the instantaneous bit rate will fluctuate with the complexity of the scene.

[0096] As a specific example, the first constant quantization parameter is 10, and the second constant quantization parameter is 40. It can be understood that the first constant quantization parameter and the second constant quantization parameter can also be other values, as long as the parameter value of the first constant quantization parameter is less than the parameter value of the second constant quantization parameter. To ensure the image quality of the reconstructed virtual viewpoint, it is recommended that the value of the first constant quantization parameter is less than or equal to 22.

[0097] In a specific implementation, the first constant quantization parameter and the second constant quantization parameter may be set based on a preset pixel block size, for example, the pixel block size is 3*3 pixels; or for another example, the pixel block size is 32*32 pixels.

[0098] In some other embodiments, the texture map region may also be kept using a constant quantization parameter, which may be the same as or different from the quantization parameter used in the non-foreground edge pixel region in the depth map region.

[0099] In another embodiment of the present specification, a constant quantization parameter can be used for the texture map area of ​​the stitched image, the non-foreground edge pixel area and the foreground edge pixel area of ​​the depth map area, wherein the quantization parameter corresponding to the foreground edge pixel area of ​​the depth map area has the smallest value. As an optional example, the values ​​of the three can be decreasing in sequence.

[0100] In other embodiments of the present specification, a preset constant quality factor (CQF) may be used to encode the ROI pixel area in the depth map area. Similar to CQP, CQF pursues constant subjectively perceived quality, and the instantaneous bit rate will also fluctuate with scene complexity.

[0101] In a specific implementation, for the stitched image, a constant bit rate can be used for rate distortion optimization. For example, a bit rate control module can be set in a compression device such as an encoder or a device including an encoder, and a series of encoding parameters can be selected to control the bit rate of the encoded video to meet specific application requirements and make the encoding distortion as small as possible. In some embodiments of the present specification, for a video frame, the bit rate coefficient of the entire stitched image can be controlled to be constant. At the same time, for the foreground edge pixel area in the depth map area, a first constant quantization parameter can be used for encoding. For the non-foreground edge pixel area in the depth map area and the texture map area, the corresponding quantization parameter value can be determined based on the bit rate coefficient of the entire stitched image and the proportion of the foreground edge pixel area in the depth map area. The obtained quantization parameter value may be fixed or may change instantaneously.

[0102] In a specific implementation, the pixels of the depth map sub-region in the stitched image may be all or part of the pixels in a set of pixels corresponding one-to-one to the pixels in the texture map sub-region of the corresponding viewpoint. Fig. 9 As shown, the pixel points of the depth map in the stitched image are all the pixel points in the pixel point set that correspond one-to-one to the pixel points in the texture map of the corresponding viewpoint. The texture map of each corresponding viewpoint can be the original texture map collected by the camera, or the texture map after the original texture map has a reduced resolution. Correspondingly, the depth map of each corresponding viewpoint can be the original depth map estimated based on the original texture map of the corresponding viewpoint, or the depth map after the original depth map has a reduced resolution.

[0103] In some embodiments of the present specification, the texture map region includes a plurality of texture map sub-regions corresponding to viewpoints, the depth map region includes a plurality of depth map sub-regions corresponding to viewpoints, and the depth map sub-region is smaller than the texture map sub-region. Fig.11 or Fig.12 The structural schematic diagram of the stitched image shown in FIG. 1 is a schematic diagram of the structure of the stitched image, wherein the texture map area includes 8 texture map sub-areas corresponding to the viewpoints, and the depth map area includes 8 depth map sub-areas corresponding to the viewpoints, and the depth map sub-areas are smaller than the texture map sub-areas. In addition, the texture map area and the depth map area in the stitched image can be divided into multiple areas, such as Fig.12As shown, the depth map area is divided into an area above the texture map and an area below the texture map, that is, the texture map area is a continuous area, and the depth map area includes two continuous areas.

[0104] like Fig.11 and Fig.12 The stitched image shown, wherein the depth map sub-region in the stitched image can be obtained by downsampling the original depth map corresponding one-to-one to the pixel points in the texture map of the corresponding viewpoint.

[0105] Understandably, Fig.11 or Fig.12 This is not a restriction on dividing the stitched image in a non-equal manner. In a specific implementation, the number of texture maps and depth maps in the stitched image is determined based on the number of synchronously collected viewpoints. The pixel quantity and aspect ratio of the stitched image can be diverse, and the division method can also be diverse.

[0106] It can be understood that, in a specific implementation, the number of pixels in the depth map area may also be greater than the number of pixels occupied by the pixel data in the texture map area.

[0107] Alternatively, see Fig.13 and Fig.14 The structural schematic diagram of the spliced ​​image is as follows: the texture map area may include two continuous areas, and the depth map area may also include two continuous areas. The texture map area and the depth map area may be arranged alternately.

[0108] Or, see Fig.15 The texture map sub-regions included in the texture map region may be arranged alternately with the depth map sub-regions included in the depth map region. The number of continuous regions included in the texture map region may be equal to the number of texture map sub-regions, and the number of continuous regions included in the depth map region may be equal to the number of depth map sub-regions.

[0109] In a specific implementation, the pixel data of each texture map may be stored in the texture map sub-region according to the order in which the pixels are arranged, and the depth data corresponding to each texture map may also be stored in the depth map sub-region according to the order in which the pixels are arranged.

[0110] Combined with reference Figures 16 to 18 , Fig.16 The texture is shown with 9 pixels. Figure 1 , Fig.17 The texture is shown with 9 pixels. Figure 2 , Texture Figure 1 and texture Figure 2 It is two images with different viewpoints (or different perspectives) synchronized. Figure 1 and texture Figure 2 , we can get the corresponding texture Figure 1 Depth data, including textures Figure 1 Depth value 1 to texture Figure 1 Depth value 9, and the corresponding texture Figure 2 Depth data, including textures Figure 2 Depth value 1 to texture Figure 2 Depth value 9.

[0111] See also Fig.18 , in the texture Figure 1 Storing in image sub-regions, the texture can be arranged in the order of pixels. Figure 1 The image sub-region stored in the upper left corner, that is, in the image sub-region, the arrangement of pixels can be consistent with the texture Figure 1 Same. Texture Figure 2 The storage to the image sub-region may also be performed in the upper right image sub-region in this manner.

[0112] Similarly, the texture Figure 1 The depth data of the depth map sub-region can be stored in a similar manner. In the case where the depth value corresponds to the pixel value of the texture map one by one, the depth data of the depth map sub-region can be stored in a similar manner as follows: Fig.18 If the depth value is obtained by downsampling the original depth map, it can be stored in the depth map sub-region according to the order of pixel arrangement of the depth map obtained after downsampling.

[0113] Those skilled in the art can understand that the compression rate of an image is related to the correlation between the pixels in the image. The stronger the correlation, the higher the compression rate. Since the captured image corresponds to the real world, the correlation between the pixels is strong. By storing the pixel data and depth data of the image in the order of pixel arrangement, the compression rate can be higher when the spliced ​​image is compressed. That is, the amount of data after compression can be smaller when the amount of data before compression is the same.

[0114] By dividing the stitched image into a texture map area and a depth map area, when multiple texture map sub-areas are adjacent in the texture map area, or multiple depth map sub-areas are adjacent in the depth map area, since the data stored in each texture map sub-area is obtained from an image (texture map) or a frame image in a video taken from different viewpoints (different angles) of the viewing area, and the data stored in the depth map area are all depth maps, a higher compression rate can also be obtained when compressing the stitched image.

[0115] In a specific implementation, edge protection may be performed on all or part of the texture map sub-region and the depth map sub-region. The edge protection may be in various forms, for example, Fig.14Taking the viewpoint 1 depth map as an example, redundant pixels can be set around the original viewpoint 1 depth map; or the number of pixels of the original viewpoint 1 depth map can be kept unchanged, and redundant pixels that do not store actual pixel data can be reserved around it, and the original viewpoint 1 depth map can be reduced and stored in the remaining pixels; or other methods can be used to ultimately leave redundant pixels between the viewpoint 1 depth map and other images around it.

[0116] Since the stitched image includes multiple texture maps and depth maps, the correlation between adjacent boundaries of each image is poor. By performing edge protection, the quality loss of the texture map and depth map in the stitched image can be reduced when the stitched image is compressed.

[0117] In a specific implementation, the pixel field of the texture map sub-area can store three-channel data, and the pixel field of the depth map sub-area can store single-channel data. The pixel field of the texture map sub-area is used to store pixel data of any one of the synchronized multiple texture maps, and the pixel data is usually three-channel data, such as RGB data or YUV data.

[0118] The depth map sub-area is used to store the depth data of the depth map. If the depth value is 8-bit binary data, it can be stored in a single channel of the pixel field. If the depth value is 16-bit binary data, it can be stored in a dual channel of the pixel field. Alternatively, the depth value can also be stored in a larger pixel area. For example, if the synchronized multiple texture maps are all 1920*1080 images and the depth value is 16-bit binary data, the depth value can also be stored in a 1920*1080 depth map area that is twice as large as the 1920*1080 depth map area, and each texture map area is stored as a single channel. The spliced ​​image can also be divided in combination with this specific storage method.

[0119] The uncompressed data volume of the stitched image is stored in a manner where each channel of each pixel occupies 8 bits, and can be calculated according to the following formula: the number of synchronized multiple texture maps * (the data volume of the pixel data of the texture map + the data volume of the pixel data of the depth map).

[0120] If the original texture image is 1080P resolution, that is, 1920*1080 pixels, in a progressive scan format, the original depth image can also occupy 1920*1080 pixels, which is a single channel. Then the pixel data volume of the original texture image is: 1920*1080*8*3bit, and the data volume of the original depth image is 1920*1080*8bit. If the number of cameras is 30, the pixel data volume of the stitched image is 30*(1920*1080*8*3+1920*1080*8)bit, which is about 237M. If it is not compressed, it will occupy more system resources and have a longer delay. Especially when the bandwidth is small, for example, when the bandwidth is 1Mbps, an uncompressed stitched image takes about 237s to transmit, the real-time performance is poor, and the user experience needs to be improved.

[0121] The amount of data for the stitched image can be reduced by one or more of the following methods: regular storage to obtain a higher compression rate, reducing the resolution of the original texture map, or using the pixel data after the resolution is reduced as the pixel data of the texture map, or downsampling one or more of the original depth maps.

[0122] For example, if the resolution of the original texture image is 4K, i.e., 4096*2160 pixel resolution, and it is downsampled to 540P, i.e., 960*540 pixel resolution, then the number of pixels of the stitched image is about one sixteenth of that before downsampling. In combination with any one or more of the above other data reduction methods, the data volume can be reduced.

[0123] It is understandable that if the bandwidth supports and the decoding capability of the device performing data processing can support stitched images with higher resolution, stitched images with higher resolution can also be generated to improve the image quality.

[0124] Those skilled in the art will appreciate that, in different application scenarios, the pixel data and depth data of multiple synchronized texture images may also be stored in other ways, for example, in units of pixels to form a spliced ​​image. Fig.16 , Fig.17 and Fig.19 ,for Fig.16 and Fig.17 Texture shown Figure 1 and texture Figure 2 , you can Fig.19 The stitched image is stored in the same way.

[0125] In summary, the pixel data of texture maps of multiple viewpoints and the depth data of corresponding viewpoint depth maps can be stored in a stitched image. The stitched image can be divided into texture map areas and depth map areas in a variety of ways, or it can be stored in a preset order without division.

[0126] In a specific implementation, the synchronized multiple texture maps may also be synchronized multiple frame images obtained by decoding multiple videos. The videos may be acquired through multiple cameras, and their settings may be the same or similar to the cameras used to acquire the texture maps in the above text.

[0127] In the above-mentioned embodiment, the storage is a method of splicing images, for example Fig. 9 and Figures 11 to 19 The storage methods illustrated can all be described by corresponding association relationship fields, so that a device performing data processing can obtain associated texture map and depth map data according to the association relationship fields.

[0128] In a specific implementation, the image format of the stitched image can be any one of the image formats such as BMP, PNG, JPEG, Webp, or other image formats. The storage method of pixel data and depth data in free viewpoint video data (multi-angle free viewpoint image data) is not limited to the method of stitching images. It can be stored in other ways, and there can also be corresponding association relationship field descriptions.

[0129] Reference Fig. 20 The schematic diagram of a video compression method for a specific application scenario is shown. For a spliced ​​image I of any free viewpoint video frame obtained, the method described in the aforementioned step S102 can be used to identify the texture map area Zt and the depth map area Zd in the spliced ​​image I. Then, for the texture map area Zt, a preset first encoding method is used for encoding to obtain a compressed texture map area Zt', and for the depth map area Zd, a preset second encoding method is used for encoding to obtain a compressed depth map area Zd', wherein the second encoding method is an encoding method based on a region of interest; based on the compressed texture map area Zt' and the compressed depth map area Zd', a compressed video frame I' can be obtained.

[0130] After the compressed video frames obtained by the above-mentioned video encoding and compression method are transmitted to the terminal side (such as a display device), they can be decompressed by a corresponding decompression device, and the free viewpoint video image can be reconstructed based on the spliced ​​image obtained after decompression.

[0131] Reference Fig.21 The flowchart of the decompression method shown in FIG. 1 shows that, for the compressed video frame obtained by the compression process in the above-mentioned embodiment, the decompression process can be performed accordingly in the following manner:

[0132] S211, obtaining a compressed video frame and quantization parameters of pixel blocks in the compressed video frame, wherein the compressed video frame includes a texture map area encoded by a first encoding method and a depth map area encoded by a second encoding method, and the second encoding method is an encoding method based on a region of interest.

[0133] The texture map area includes texture maps of multiple synchronized viewpoints, and the depth map area includes depth maps of viewpoints corresponding to each texture map.

[0134] S212, based on the quantization parameters of the pixel blocks in the compressed video frame, decompress the compressed video frame according to the pixel blocks to obtain a spliced ​​image of the free viewpoint video frame, wherein the spliced ​​image includes the texture map areas of the synchronized multiple viewpoints and the depth map areas of the corresponding viewpoints.

[0135] Since the texture map area and the depth map area in the compressed video frame adopt different encoding methods, the quantization parameters in the corresponding pixel blocks will be different. Therefore, based on the quantization parameters of the pixel blocks in the compressed video frame, the compressed video frame can be decompressed according to the pixel blocks, so as to obtain a spliced ​​image of the free viewpoint video frame.

[0136] The compressed video frame is decompressed in the above-mentioned manner. Since the compressed video frame includes a texture map area encoded by a first encoding method and a depth map area encoded by a second encoding method, the second encoding method is an encoding method based on a region of interest, and can be compressed according to image features of the depth map area. Therefore, the depth map area obtained after decompression has a higher image quality, so the image quality of the free viewpoint video reconstructed based on the depth map can be improved.

[0137] After obtaining the free viewpoint video frame using the method of the above embodiment, the image reconstruction of the virtual viewpoint can be achieved based on the acquired free viewpoint and parameter data corresponding to the free viewpoint video frame and based on the texture map and depth map of the corresponding viewpoint in the free viewpoint video frame.

[0138] In a specific implementation, based on the free viewpoint, the texture map and the corresponding depth map of some viewpoints in the free viewpoint video frame can be selected, and the texture map and the corresponding depth map of the selected partial viewpoint are used for combined rendering to obtain a reconstructed image of the virtual viewpoint.

[0139] In some embodiments of the present specification, it can be determined based on the spatial position relationship between the virtual viewpoint and each viewpoint in the free viewpoint video frame, and specifically can be determined based on parameter data corresponding to the free viewpoint video frame.

[0140] In a specific implementation, in order to enrich the user's visual experience, augmented reality (AR) special effects can be embedded in the reconstructed free viewpoint image. To implement the embedding of augmented reality special effects, it is necessary to determine the special effect rendering pixel area in the texture map area and the virtual information image to be embedded.

[0141] Among them, the special effect rendering pixel area can be generated based on user interaction, or determined based on certain preset trigger conditions or third-party instructions.

[0142] The virtual information image to be implanted can be provided directly or generated in real time based on interaction. After determining the special effect rendering pixel area, the corresponding augmented reality special effect input data can be obtained to generate a matching virtual information image.

[0143] In a specific implementation, a special effect rendering pixel area can be obtained by three-dimensional calibration, and then a virtual information image matching the special effect rendering pixel area can be obtained. To calibrate the special effect rendering pixel area, the position of the virtual rendering target object corresponding to the special effect rendering pixel area in the reconstructed image can be determined.

[0144] By calibrating, a virtual information image that matches the position of the virtual rendering object in the special effects rendering pixel area is obtained, so that the obtained virtual information image can be more closely matched with the position of the virtual rendering target object in the three-dimensional space, and the displayed virtual information image is more consistent with the real state in the three-dimensional space, so that the displayed composite image is more realistic and vivid, enhancing the user's visual experience.

[0145] In a specific implementation, based on the augmented reality special effect input data of the virtual rendering target object, a virtual information image corresponding to the target object can be generated according to a preset special effect generation method. Specifically, a variety of special effect generation methods can be used.

[0146] For example, the augmented reality special effect input data of the target object may be input into a preset three-dimensional model, and based on the position of the virtual rendering target object in the image obtained by three-dimensional calibration, a virtual information image matching the virtual rendering target object may be output;

[0147] For example, the augmented reality special effects input data of the virtual rendering target object can be input into a preset machine learning model, and based on the position of the virtual rendering target object in the image obtained by three-dimensional calibration, a virtual information image matching the virtual rendering target object is output.

[0148] In an embodiment of the present specification, in the video compression process, in order to avoid compression loss of the texture map area in the area where the special effects are embedded, the texture map area can be coded and compressed based on ROI. In an embodiment of the present specification, for the special effect rendering pixel area in the texture map area, a special coding method suitable for augmented reality special effects is used for coding.

[0149] The embodiments of this specification also provide embodiments of corresponding video compression devices, such as Fig. 22 The schematic diagram of the structure of the video compression device shown in the figure, wherein the video compression device 220 may include: a video frame acquisition unit 221, an identification unit 222, a first encoding unit 223 and a second encoding unit 224, wherein:

[0150] The video frame acquisition unit 221 is adapted to acquire a free viewpoint video frame, wherein the video frame includes a spliced ​​image formed by texture maps of multiple viewpoints and depth maps of corresponding viewpoints;

[0151] The identification unit 222 is adapted to identify a texture map region and a depth map region in the stitched image;

[0152] The first encoding unit 223 encodes the texture map region using a preset first encoding method;

[0153] The second encoding unit 224 encodes the depth map region by adopting a preset second encoding method, where the second encoding method is an encoding method based on a region of interest.

[0154] By using the above-mentioned video compression device, different encoding methods are used to compress the texture map area and the depth map area in the stitched image of the video frame, wherein the depth map area is compressed using an encoding method based on the region of interest, and can be compressed according to the image features of the depth map area, thereby reducing the compression loss of the depth map, and further improving the image quality of the free viewpoint video reconstructed based on the depth map.

[0155] In a specific implementation, the video compression device can be implemented as a hardware encoder, an encoding chip or an encoding module composed of multiple modules. The hardware encoder, encoding chip or encoding module can be equipped with a special hardware logic device and / or software algorithm to compress the acquired free viewpoint video frame to obtain a compressed video frame.

[0156] The embodiment of this specification also provides a video decompression device corresponding to the video compression device, referring to Fig.23 The structural diagram of the video decompression device shown in FIG. Fig.23 As shown, the video decompression device 230 may include an acquisition unit 231 and a decompression unit 232, wherein:

[0157] The acquisition unit 231 is adapted to acquire a compressed video frame and a quantization parameter of a pixel block in the compressed video frame, wherein the compressed video frame includes a texture map area encoded by a first encoding method and a depth map area encoded by a second encoding method, wherein the texture map area includes texture maps of multiple synchronized viewpoints, the depth map area includes depth maps of viewpoints corresponding to each texture map, and the second encoding method is an encoding method based on a region of interest;

[0158] The decompression unit 232 is suitable for decompressing the compressed video frame according to the pixel blocks based on the quantization parameters of the pixel blocks in the compressed video frame to obtain a spliced ​​image of the free viewpoint video frame, wherein the spliced ​​image includes the texture map area of ​​the synchronized multiple viewpoints and the depth map area of ​​the corresponding viewpoints.

[0159] In a specific implementation, the video decompression device can be implemented as a hardware decoder, a decoding chip or a decoding module composed of multiple modules. The hardware decoder, decoding chip or decoding module can be provided with a special hardware logic device and / or software algorithm to decompress the acquired free viewpoint compressed video frame to obtain a free viewpoint video frame.

[0160] In the embodiments of the present specification, in the process of video compression or decompression, in addition to automatically selecting the encoding mode or decoding mode based on the recognition result of the system, the specific encoding mode and decoding mode can also be selected based on user interaction. Specifically, in response to the encoding selection interaction operation, an encoding selection instruction can be generated, which can trigger the corresponding encoding unit (such as a specific encoder, encoding module, encoding chip, encoding software, encoding system, etc., one or more thereof) to perform encoding; similarly, in response to the decoding selection interaction operation, a decoding selection instruction can be generated, which can trigger the corresponding decoding unit (such as a specific decoder, decoding module, decoding chip, decoding software, decoding system, etc., one or more thereof) to perform decoding.

[0161] The present specification also provides an electronic device, referring to Fig.24 The structural diagram of the electronic device shown in the figure, wherein the electronic device 240 may include a memory 241 and a processor 242, the memory 241 stores computer instructions that can be executed on the processor 242, wherein the processor can execute the steps of the method described in any of the aforementioned embodiments when running the computer instructions.

[0162] Based on the location of the electronic device in the entire video processing system, the electronic device may further include other electronic components or assemblies.

[0163] For example, continue to refer to Fig.24The electronic device 240 may also include a communication component 243, which can communicate with the acquisition system or the cloud server to obtain a synchronized texture map of multiple viewpoints for generating free viewpoint video frames, or obtain a free viewpoint compressed video frame obtained after compression processing using the method of the aforementioned embodiment of this specification. The processor 242 can then perform decompression processing based on the compressed video frame obtained by the communication component 243, and perform free viewpoint video reconstruction according to the virtual viewpoint position.

[0164] For example, in some electronic devices, continue to refer to Fig.24 The electronic device 240 may further include a display component 244 (eg, a display, a touch screen, a projector) to display the reconstructed image of the virtual viewpoint.

[0165] As a server device, the electronic device can be set as a cloud server or server cluster, or as a local server to compress the free viewpoint video frame before transmission. As a terminal device, the electronic device can be a handheld electronic device such as a mobile phone, a laptop computer, a desktop computer, a set-top box, and other electronic devices with video processing and playback functions. The terminal device can be used to decompress the received compressed video frame, and reconstruct the image of the virtual viewpoint based on the decompressed video frame and the acquired virtual viewpoint. The virtual viewpoint can be determined based on user interaction behavior or based on pre-set virtual viewpoint position information.

[0166] In some embodiments of the present specification, the memory, processor, communication component, and display component may communicate with each other via a bus network.

[0167] In the specific implementation, Fig.24 As shown, the communication component 243 and the display component 244 etc. may be components disposed inside the electronic device 240, or may be external devices connected via expansion components such as an expansion interface, an expansion dock, and an expansion cable.

[0168] In a specific implementation, the processor 242 can be implemented collaboratively through any one or more of a central processing unit (CPU) (such as a single-core processor, a multi-core processor), a CPU group, a graphics processing unit (GPU), an AI chip, an FPGA chip, etc.

[0169] In a specific implementation, for a large number of free viewpoint video frames, in order to reduce processing delay, an electronic device cluster consisting of multiple electronic devices can also be used for collaborative implementation.

[0170] In order to enable those skilled in the art to better understand and implement, a specific application scenario is used below for illustration. Fig.25 The structural diagram of the video processing system shown in FIG. Fig.25 As shown, it is a structural diagram of a video processing system in an application scenario, wherein a layout scenario of a data processing system for a basketball game is shown. The video processing system A0 includes an acquisition array A1 composed of multiple acquisition devices, a data processing device A2, a cloud server cluster A3, a playback control device A4, a playback terminal A5 and an interactive terminal A6.

[0171] Reference Fig.25 , with the basketball hoop on the left as the core viewing point, and the fan-shaped area with the core viewing point as the center of the circle and located on the same plane as the core viewing point as the preset multi-angle free viewing angle range. The acquisition devices in the acquisition array A1 can be placed in different positions of the on-site acquisition area in a fan-shaped manner according to the preset multi-angle free viewing angle range, and can respectively and simultaneously acquire video data streams from corresponding angles in real time.

[0172] In a specific implementation, the acquisition equipment can also be set in the ceiling area of ​​a basketball stadium, on a basketball stand, etc. The acquisition equipment can be arranged and distributed along a straight line, a fan, an arc, a circle or an irregular shape. The specific arrangement can be set according to one or more factors such as the specific on-site environment, the number of acquisition equipment, the characteristics of the acquisition equipment, the imaging effect requirements, etc. The acquisition equipment can be any device with a camera function, such as an ordinary camera, a mobile phone, a professional camera, etc.

[0173] In order not to affect the work of the acquisition equipment, the data processing device A2 can be placed in a non-acquisition area on site and can be regarded as an on-site server. The data processing device A2 can send a stream pulling instruction to each acquisition device in the acquisition array A1 through a wireless local area network. Based on the stream pulling instruction sent by the data processing device A2, each acquisition device in the acquisition array A1 transmits the obtained video data stream to the data processing device A3 in real time. Among them, each acquisition device in the acquisition array A1 can transmit the obtained video data stream to the data processing device A2 in real time through the switch A7. The acquisition array A1 and the switch A7 together form an acquisition system.

[0174] When the data processing device A2 receives the video frame capture instruction, it captures the video frame at the specified frame time from the received multi-channel video data streams to obtain frame images of multiple synchronous video frames, and uploads the obtained multiple synchronous video frames at the specified frame time to the server cluster A3 in the cloud.

[0175] Correspondingly, the server cluster A3 in the cloud takes the original texture maps of the received multiple synchronized video frames as an image combination, determines the parameter data corresponding to the image combination and the estimated depth map corresponding to each original texture map in the image combination, and based on the parameter data corresponding to the image combination, the pixel data of the texture map in the image combination and the depth data of the corresponding depth map, reconstructs the frame image based on the acquired virtual viewpoint path to obtain corresponding multi-angle free-viewpoint video data.

[0176] As a depth map post-processing step, the depth map correction method introduced in the previous embodiment of this specification can be used to perform depth map correction on the estimated depth map of the corresponding viewpoint.

[0177] The server may be placed in the cloud, and in order to process data in parallel more quickly, a server cluster A3 in the cloud may be formed by a plurality of different servers or server groups according to different data to be processed.

[0178] For example, the cloud server cluster A3 may include: a first cloud server A31, a second cloud server A32, a third cloud server A33, and a fourth cloud server A34. The first cloud server A31 may be used to determine the parameter data corresponding to the image combination; the second cloud server A32 may be used to determine the estimated depth map of the original texture map of each viewpoint in the image combination and perform depth map correction processing; the third cloud server A33 may perform frame image reconstruction based on the position information of the virtual viewpoint, the parameter data corresponding to the image combination, the texture map and the depth map of the image combination, and the depth image based on the depth image, using a virtual viewpoint reconstruction (Depth Image Based Rendering, DIBR) algorithm to obtain an image of the virtual viewpoint; the fourth cloud server A34 may be used to generate free viewpoint video (multi-angle free viewpoint video).

[0179] It is understandable that the first cloud server A31, the second cloud server A32, the third cloud server A33, and the fourth cloud server A34 may also be a server group composed of a server array or a server sub-cluster, and the embodiment of the present invention does not limit this.

[0180] Then, the playback control device A4 can insert the received free viewpoint video frame into the video stream to be played, and the playback terminal A5 receives the video stream to be played from the playback control device A4 and plays it in real time. Among them, the playback control device 34 can be a manual playback control device or a virtual playback control device. In a specific implementation, a special server that can automatically switch video streams can be set as a virtual playback control device to control the data source. A director control device such as a director station can be used as a playback control device in an embodiment of the present invention.

[0181] It can be understood that the data processing device A2 can be placed in a non-collection area on site or in the cloud according to the specific scenario, and the server (cluster) and playback control device can be placed in a non-collection area on site, in the cloud or on the terminal access side according to the specific scenario. This embodiment is not used to limit the specific implementation and protection scope of the present invention.

[0182] The embodiments of the present specification also provide a computer-readable storage medium on which computer instructions are stored, wherein the computer instructions, when executed, execute the steps of the method described in any of the aforementioned embodiments. For details, please refer to the introduction of the aforementioned embodiments, which will not be repeated here.

[0183] In a specific implementation, the computer-readable storage medium may be any appropriate readable storage medium such as an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0184] Although the embodiments of this specification are disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this specification, so the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A video compression method, wherein: include: Acquire a free viewpoint video frame, wherein the video frame includes a spliced ​​image formed by texture maps of multiple synchronized viewpoints and depth maps of corresponding viewpoints; Identifying a texture map region and a depth map region in the stitched image; For the texture map area, a preset first encoding method is used for encoding, and for the depth map area, a preset second encoding method is used for encoding, so as to obtain a compressed video frame; wherein the second encoding method is an encoding method based on a region of interest, and the depth map area is compressed using an encoding method based on a region of interest, so as to compress according to image features of the depth map area; The texture map area includes a special effect rendering pixel area; the texture map area is encoded using a preset first encoding method, including: For the special effect rendering pixel area in the texture map area, encoding is performed using an encoding method suitable for augmented reality special effects; The encoding of the depth map area by using a preset second encoding method includes: The ROI pixel area in the depth map area is encoded based on a preset constant quantization parameter, or encoded using a preset constant quality factor.

2. The method according to claim 1, wherein: The encoding of the ROI pixel area in the depth map area based on a preset constant quantization parameter includes: A first constant quantization parameter is used to encode the foreground edge pixel area in the depth map sub-area corresponding to each viewpoint contained in the depth map area, and a second constant quantization parameter is used to encode the non-foreground edge pixel area in the depth map sub-area corresponding to each viewpoint, and the parameter value of the first constant quantization parameter is less than the parameter value of the second constant quantization parameter.

3. The method according to any one of claims 1 to 2, wherein: Also includes: For the stitched image, a constant bit rate is adopted.

4. The method according to any one of claims 1 to 2, wherein: The pixel points of the depth map sub-region in the stitched image are all or part of the pixel points in a set of pixel points that correspond one-to-one to the pixel points in the texture map sub-region of the corresponding viewpoint.

5. The method according to claim 4, wherein: The texture map region includes a plurality of texture map sub-regions corresponding to viewpoints, and the depth map region includes a plurality of depth map sub-regions corresponding to viewpoints, wherein the depth map sub-regions are smaller than the texture map sub-regions.

6. The method according to claim 5, wherein: The depth map sub-region in the stitched image is obtained by downsampling the original depth map corresponding one-to-one to the pixel points in the texture map of the corresponding viewpoint.

7. A video decompression method, wherein: include: Obtaining a compressed video frame and a quantization parameter of a pixel block in the compressed video frame, wherein the compressed video frame includes a texture map area encoded by a first encoding method and a depth map area encoded by a second encoding method, wherein the texture map area includes texture maps of multiple synchronized viewpoints, the depth map area includes depth maps of viewpoints corresponding to each texture map, and the second encoding method is an encoding method based on a region of interest; The texture map area includes a special effect rendering pixel area; the texture map area is encoded using a preset first encoding method, including: For the special effect rendering pixel area in the texture map area, encoding is performed using an encoding method suitable for augmented reality special effects; For the depth map area, encoding is performed using a preset second encoding method, including: For the ROI pixel area in the depth map area, encoding is performed based on a preset constant quantization parameter, or encoding is performed using a preset constant quality factor; Based on the quantization parameters of the pixel blocks in the compressed video frame, the compressed video frame is decompressed according to the pixel blocks to obtain a spliced ​​image of the free viewpoint video frame, wherein the spliced ​​image includes the texture map areas of the synchronized multiple viewpoints and the depth map areas of the corresponding viewpoints.

8. A video compression device, wherein: include: A video frame acquisition unit, adapted to acquire a free viewpoint video frame, wherein the video frame includes a spliced ​​image formed by texture maps of multiple viewpoints and depth maps of corresponding viewpoints; an identification unit, adapted to identify a texture map region and a depth map region in the stitched image; A first encoding unit, encoding the texture map region by using a preset first encoding method; A second encoding unit, for encoding the depth map region using a preset second encoding method, wherein the second encoding method is an encoding method based on a region of interest, wherein the depth map region is compressed using the encoding method based on the region of interest, so as to be compressed according to image features of the depth map region; The texture map area includes a special effect rendering pixel area; the texture map area is encoded using a preset first encoding method, including: For the special effect rendering pixel area in the texture map area, encoding is performed using an encoding method suitable for augmented reality special effects; For the depth map area, encoding is performed using a preset second encoding method, including: The ROI pixel area in the depth map area is encoded based on a preset constant quantization parameter, or encoded using a preset constant quality factor.

9. A video decompression device, wherein: include: an acquisition unit, adapted to acquire a compressed video frame and a quantization parameter of a pixel block in the compressed video frame, wherein the compressed video frame includes a texture map region encoded by a first encoding method and a depth map region encoded by a second encoding method, wherein the texture map region includes texture maps of multiple synchronized viewpoints, the depth map region includes depth maps of viewpoints corresponding to each texture map, and the second encoding method is an encoding method based on a region of interest; The texture map area includes a special effect rendering pixel area; the texture map area is encoded using a preset first encoding method, including: For the special effect rendering pixel area in the texture map area, encoding is performed using an encoding method suitable for augmented reality special effects; For the depth map area, encoding is performed using a preset second encoding method, including: For the ROI pixel area in the depth map area, encoding is performed based on a preset constant quantization parameter, or encoding is performed using a preset constant quality factor; A decompression unit is adapted to decompress the compressed video frame according to the pixel blocks based on the quantization parameters of the pixel blocks in the compressed video frame to obtain a spliced ​​image of the free viewpoint video frame, wherein the spliced ​​image includes the texture map area of ​​the synchronized multiple viewpoints and the depth map area of ​​the corresponding viewpoints.

10. An electronic device comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, wherein: When the processor executes the computer instructions, the steps of the method according to any one of claims 1 to 6 or claim 7 are performed.

11. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed, the steps of the method described in any one of claims 1 to 6 or claim 7 are executed.

Images