Video processing method, device, electronic device and storage medium

Abstract

A video processing method, device, electronic device and storage medium, wherein a video processing method comprises: splicing texture maps of multiple viewpoints of frame synchronization into a texture map splicing image; splicing depth maps corresponding to the texture maps of multiple viewpoints of frame synchronization into a depth map splicing image; encoding the texture map splicing image and the depth map splicing image respectively to obtain a corresponding texture map compressed image and depth map compressed image. The above scheme can reduce the requirements on the decoding capability of the decoding end and reduce compression loss.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of video processing technology, and more particularly to a video processing method, device, electronic device, and storage medium. Background Art

[0002] Video data is data that supports video playback for users to watch. Usually, video data only supports users to watch from one perspective, and users cannot adjust the viewing angle. Free viewpoint video is a technology that can provide a high degree of freedom viewing experience. Users can adjust the viewing angle through interactive operations during the viewing process and watch from the free viewpoint they want, which can greatly improve the viewing experience.

[0003] In order to realize the storage, transmission and playback of free viewpoint video data, the video images need to be encoded and packaged. The commonly used encoding method is: splicing the texture maps of multiple synchronized viewpoints and the corresponding depth maps to obtain a spliced ​​image, and then encoding the spliced ​​image in a unified manner.

[0004] However, the above encoding method has many limitations. For example, the above encoding method may cause compression loss to the spliced ​​image, thereby affecting the image quality of the reconstructed free viewpoint video; for another example, due to the high resolution of the spliced ​​image, the decoding capability of the decoding end may not meet the requirements. Summary of the invention

[0005] In view of this, the embodiments of the present specification provide a video processing method, an apparatus, an electronic device, and a storage medium, which can reduce the requirements on the decoding capability of the decoding end and reduce the compression loss.

[0006] The embodiment of this specification provides a video processing method, including:

[0007] splicing texture maps of multiple viewpoints synchronized by frames into a texture map splicing image;

[0008] splicing the depth maps corresponding to the texture maps of the multiple viewpoints of the frame synchronization into a depth map splicing image;

[0009] The texture map stitched image and the depth map stitched image are encoded respectively to obtain a corresponding texture map compressed image and depth map compressed image.

[0010] Optionally, the method further includes: encapsulating the texture map compressed image and the depth map compressed image in a video compression frame of the same video channel.

[0011] Optionally, the depth map compressed image is encapsulated in a frame header area of ​​the video compression frame.

[0012] Optionally, the depth map stitching image is encoded using a region of interest based encoding method to obtain a corresponding depth map compressed image.

[0013] Optionally, the ROI pixel area in the depth map stitched image is encoded based on a preset constant quantization parameter to obtain a corresponding depth map compressed image.

[0014] Optionally, a first constant quantization parameter is used to encode the foreground edge pixel area in the depth map sub-area corresponding to each viewpoint contained in the depth map stitched image, and a second constant quantization parameter is used to encode the non-foreground edge pixel area in the depth map sub-area corresponding to each viewpoint, and the parameter value of the first constant quantization parameter is less than the parameter value of the second constant quantization parameter.

[0015] Optionally, the pixel points of the depth map sub-region corresponding to each viewpoint contained in the depth map stitched image are all or part of the pixel points in a set of pixel points in the texture map stitched image that correspond one-to-one to the pixel points in the texture map sub-region of the corresponding viewpoint.

[0016] The embodiment of this specification also provides another video processing method, including:

[0017] Determine the coding category to which the coding method belongs;

[0018] If it is determined that the encoding method belongs to custom encoding of the depth map, the encoding identification information corresponding to the encoding method is stored in the encoding information area in the image set user information; and the texture maps of multiple viewpoints synchronized with the frames are spliced ​​into a texture map spliced ​​image, and the depth maps corresponding to the texture maps of the multiple viewpoints synchronized with the frames are spliced ​​into a depth map spliced ​​image, and the texture map spliced ​​image and the depth map spliced ​​image are respectively encoded to obtain the corresponding texture map compressed image and depth map compressed image.

[0019] Optionally, the method further comprises:

[0020] The texture map compressed image and the depth map compressed image are encapsulated in the same video channel.

[0021] Optionally, encapsulating the texture map compressed image and the depth map compressed image in the same video channel includes:

[0022] The depth map compressed image is encapsulated into a depth map region in a frame header of a video compression frame of the same video channel.

[0023] Optionally, the method further comprises:

[0024] If it is determined that the encoding method belongs to multi-view joint encoding, the viewpoint identifier and frame identifier of the texture map and depth map of the multiple viewpoints synchronized with the frame are obtained and stored in the image information area in the image set user information, and the encoding identifier information corresponding to the encoding method is stored in the encoding information area in the image set user information; and the texture map and depth map of the multiple viewpoints synchronized with the frame are respectively encoded using a preset multi-view joint encoding method to obtain a texture map compressed image and a depth map compressed image.

[0025] Optionally, the method further comprises:

[0026] The texture map compressed image and the depth map compressed image are combined as an image and encapsulated into the same video channel to obtain a video compression frame including the texture map compressed image and the depth map compressed image.

[0027] Optionally, the acquiring viewpoint identifiers and frame identifiers of the texture maps and depth maps of the multiple viewpoints of the frame synchronization and storing them in an image information area in the image set user information, and storing the encoding identifier information corresponding to the encoding method in an encoding information area in the image set user information includes:

[0028] The coding category identifier corresponding to the coding mode is stored in the coding category area in the channel header of the video channel;

[0029] The coding identification information is stored in the coding information area in the video compression frame header;

[0030] The viewpoint identifiers and frame identifiers of the texture maps and depth maps of the multiple viewpoints of the frame synchronization are stored in the image information area in the frame header of the video compression frame according to the acquisition order.

[0031] Optionally, the step of stitching the texture maps of the multiple viewpoints of frame synchronization into a texture map stitching image, and stitching the depth maps corresponding to the texture maps of the multiple viewpoints of frame synchronization into a depth map stitching image includes:

[0032] Scanning the texture maps of the multiple viewpoints of the frame synchronization in a raster scanning manner, and splicing the texture maps of the multiple viewpoints in a raster scanning order to obtain the texture map splicing image;

[0033] The depth maps corresponding to the texture maps of the multiple viewpoints of the frame synchronization are scanned in the raster scanning manner, and the depth maps of the multiple viewpoints are spliced ​​in a raster scanning order to obtain the depth map spliced ​​image.

[0034] The embodiment of this specification also provides another video processing method, including:

[0035] Decapsulating the free viewpoint video stream to obtain video compression frames and encoding category information of the encoding method of the video compression frames;

[0036] Based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts custom depth map encoding, the texture map compressed image and the depth map compressed image in the video compression frame are decoded respectively to obtain a texture map stitching image and a depth map stitching image; wherein the texture map stitching image includes texture maps of multiple synchronized viewpoints; and the depth map stitching image includes depth maps corresponding to the texture maps of the multiple viewpoints.

[0037] Optionally, respectively decoding the texture map compressed image and the depth map compressed image in the video compression frame to obtain the texture map stitched image and the depth map stitched image includes:

[0038] Decoding a frame header of the video compression frame, and acquiring coding identification information of the texture map compression image and coding identification information of the depth map compression image from a coding information storage area in the frame header;

[0039] Decoding the depth map compressed image using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain the depth map stitched image;

[0040] The texture map compressed image is decoded using a decoding method corresponding to the coding identification information of the texture map compressed image to obtain the texture map spliced ​​image.

[0041] Optionally, respectively decoding the texture map compressed image and the depth map compressed image in the video compression frame to obtain the texture map stitched image and the depth map stitched image includes:

[0042] Decoding the texture map compressed image using a first decoding resource to obtain a texture map spliced ​​image;

[0043] The depth map compressed image is decoded using a second decoding resource to obtain a depth map stitched image.

[0044] Optionally, the method further comprises:

[0045] Acquire parameter data of a viewpoint from a frame header of the video compression frame;

[0046] Acquire a virtual viewpoint, and based on the parameter data of the viewpoint, select a texture map in a corresponding texture map sub-region and a depth map in a depth map sub-region in the texture map stitched image and the depth map stitched image that match the position of the virtual viewpoint as a reference texture map and a reference depth map;

[0047] Based on the reference texture map and the reference depth map, an image of the virtual viewpoint is reconstructed.

[0048] Optionally, the method further comprises:

[0049] Based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts multi-view joint encoding, the texture map compressed image and the depth map compressed image in the video compression frame are decoded using the corresponding multi-view joint decoding method to obtain synchronized texture maps of multiple viewpoints and corresponding depth maps.

[0050] Optionally, decoding the joint compressed image in the video compression frame to obtain texture maps of multiple synchronized viewpoints and corresponding depth maps includes:

[0051] Decoding the frame header of the video compression frame, and acquiring the coding identification information of the texture map compression image and the coding identification of the depth map compression image from the coding information storage area in the frame header;

[0052] The texture map compressed image and the depth map compressed image are decoded using a multi-view joint decoding method corresponding to the coding identification information of the texture map compressed image and the coding identification of the depth map compressed image to obtain the texture maps of the synchronized multiple viewpoints and the corresponding depth maps.

[0053] Optionally, the method further comprises:

[0054] Obtaining the viewpoint identifier, the frame identifier of the video compression frame, and parameter data of each viewpoint from the frame header of the video compression frame;

[0055] Acquire a virtual viewpoint, and select, based on the identifier and parameter data of each viewpoint, a texture map and a depth map matching the virtual viewpoint from the texture maps and corresponding depth maps of the synchronized multiple viewpoints as a reference texture map and a reference depth map;

[0056] Based on the reference texture map and the reference depth map, an image of the virtual viewpoint is reconstructed.

[0057] The embodiment of this specification also provides another video processing method, including:

[0058] Decoding a frame header of a video compression frame, and obtaining, from the frame header, coding identification information of a depth map compression image and coding identification information of a texture map compression image included in the video compression frame;

[0059] Decoding the texture map compressed image using a decoding method corresponding to the coding identification information of the texture map compressed image to obtain a texture map spliced ​​image, wherein the texture map spliced ​​image includes texture maps of multiple synchronized viewpoints;

[0060] The depth map compressed image is decoded using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain a depth map stitching image, wherein the depth map stitching image includes depth maps corresponding to the texture map viewpoints of the multiple viewpoints.

[0061] The embodiment of this specification also provides a video processing device, including:

[0062] A texture map stitching unit, adapted to stitch the texture maps of multiple viewpoints of frame synchronization into a texture map stitching image;

[0063] A depth map stitching unit, adapted to stitch the depth maps corresponding to the texture maps of the plurality of viewpoints of the frame synchronization into a depth map stitching image;

[0064] A first encoding unit, adapted to encode the texture map splicing image to obtain a corresponding texture map compressed image;

[0065] The second encoding unit is adapted to encode the depth map stitching image to obtain a corresponding depth map compressed image.

[0066] The embodiment of this specification also provides another video processing device, including:

[0067] A coding category determination unit, adapted to determine the coding category to which the coding method belongs;

[0068] The first encoding processing unit is suitable for storing the encoding identification information corresponding to the encoding method into the encoding information area in the image set user information when it is determined that the encoding method belongs to the depth map custom encoding; and stitching the texture maps of multiple viewpoints of frame synchronization into a texture map stitching image, stitching the depth maps corresponding to the texture maps of multiple viewpoints of frame synchronization into a depth map stitching image, and encoding the texture map stitching image and the depth map stitching image respectively to obtain the corresponding texture map compressed image and depth map compressed image.

[0069] Optionally, the device further comprises:

[0070] The encapsulation unit is adapted to encapsulate the texture map compressed image and the depth map compressed image in the same video channel.

[0071] The embodiment of this specification also provides another video processing device, including:

[0072] A decapsulation unit, adapted to decapsulate the free viewpoint video stream to obtain a video compression frame and encoding category information to which the encoding method of the video compression frame belongs;

[0073] The first decoding unit is adapted to decode the texture map compressed image and the depth map compressed image in the video compression frame respectively based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts custom depth map encoding, to obtain a texture map stitching image and a depth map stitching image; wherein the texture map stitching image includes texture maps of multiple synchronized viewpoints; and the depth map stitching image includes depth maps corresponding to the texture maps of the multiple viewpoints.

[0074] Optionally, the device further comprises:

[0075] The second decoding unit is suitable for decoding the joint compressed image in the video compression frame using the corresponding multi-view joint decoding method based on the encoding category information of the video compression frame to obtain texture maps of multiple synchronized viewpoints and corresponding depth maps.

[0076] The embodiment of this specification also provides another video processing device, including:

[0077] an indication information decoding unit, adapted to decode a frame header of the video compression frame, and obtain, from the frame header, encoding identification information of the depth map compression image and encoding identification information of the texture map compression image included in the video compression frame;

[0078] A texture map decoding unit, adapted to decode the texture map compressed image in a decoding manner corresponding to the coding identification information of the texture map compressed image to obtain a texture map spliced ​​image, wherein the depth map spliced ​​image includes texture maps of multiple synchronized viewpoints;

[0079] The depth map decoding unit is adapted to decode the depth map compressed image in a decoding manner corresponding to the encoding identification information of the depth map compressed image to obtain a depth map stitching image, wherein the depth map stitching image includes depth maps corresponding to the texture map viewpoints of the multiple viewpoints.

[0080] The embodiment of this specification also provides an electronic device, including:

[0081] An image processing device adapted to stitch texture maps of multiple viewpoints of frame synchronization into a texture map stitched image, and stitch depth maps corresponding to the texture maps of multiple viewpoints of frame synchronization into a depth map stitched image;

[0082] A first encoding device, adapted to encode the texture map splicing image to obtain a corresponding texture map compressed image;

[0083] The second encoding device is adapted to encode the depth map spliced ​​image to obtain a corresponding depth map compressed image.

[0084] The embodiment of this specification also provides another electronic device, including: a first decoding device and a second decoding device, wherein:

[0085] The first decoding device is adapted to decode a frame header of a video compression frame, and when the encoding identification information obtained from the image set user information of the frame header includes the encoding identification information of a depth map compression image and the encoding identification information of a texture map compression image, the texture map compression image is decoded using a decoding method corresponding to the encoding identification information of the texture map compression image to obtain a texture map splicing image, wherein the texture map splicing image includes texture maps of multiple synchronized viewpoints, and the second decoding device is triggered to decode the depth map compression image;

[0086] The second decoding device is suitable for decoding the depth map compressed image using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain a depth map stitching image, wherein the depth map stitching image includes a depth map corresponding to the texture map viewpoint of the multiple viewpoints.

[0087] An embodiment of the present specification also provides another electronic device, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, wherein the processor executes the steps of the method described in any of the above embodiments when executing the computer instructions.

[0088] The embodiments of the present specification also provide a computer-readable storage medium on which computer instructions are stored, wherein the computer instructions, when executed, execute the steps of the method described in any of the aforementioned embodiments.

[0089] Compared with the prior art, the technical solution of the embodiment of this specification has the following beneficial effects:

[0090] On the one hand, the video processing method of the embodiment of the present specification is adopted, by stitching the texture maps of multiple viewpoints with frame synchronization into a texture map stitching image, stitching the depth maps corresponding to the texture maps of the multiple viewpoints with frame synchronization into a depth map stitching image, and encoding the texture map stitching image and the depth map stitching image respectively, to obtain the corresponding texture map compressed image and depth map compressed image. Since the texture map stitching image and the depth map stitching image are encoded respectively in this video processing process, the corresponding encoding method can be used for encoding and compression based on the features of the texture map and the depth map, thereby reducing the compression loss, and further improving the image quality of the free viewpoint video reconstructed based on the texture map stitching image and the depth map stitching image.

[0091] Furthermore, by encapsulating the texture map compressed image and the depth map compressed image in the same video channel, that is, the texture map compressed image and the depth map compressed image can reuse one video channel, so that the hardware interface of the existing hardware device can be reused, and the specifications of the existing open source framework are complied with, and the depth map and texture map are also easily synchronized.

[0092] The depth map compressed image is encapsulated in the frame header area of ​​the video compression frame. When different decoding resources are used at the decoding end to decode the texture map compressed image and the depth map compressed image respectively, the reading of irrelevant data in the depth map compressed image decoding process can be reduced, thereby improving data processing efficiency and saving processing resources.

[0093] Furthermore, the depth map stitched image is encoded using a coding method based on the region of interest, so that it can be compressed and encoded according to the image features of the depth map stitched image, thereby reducing the compression loss of the depth map and further improving the image quality of the free viewpoint video reconstructed based on the depth map.

[0094] Furthermore, the ROI pixel region in the depth map stitched image is encoded based on a preset constant quantization parameter to obtain a corresponding depth map compressed image, which can reduce the compression loss of the ROI pixel region.

[0095] Furthermore, a first constant quantization parameter is used to encode the foreground edge pixel area in the depth map sub-area corresponding to each viewpoint contained in the depth map stitched image, and a second constant quantization parameter is used to encode the non-foreground edge pixel area in the depth map sub-area corresponding to each viewpoint. Since the foreground edge pixel area in the depth map sub-area in the depth map stitched image is very critical to the reconstruction quality of the free viewpoint video, the parameter value of the first constant quantization parameter used for the foreground edge pixel area in the depth map sub-area is smaller than the parameter value of the second quantization parameter used for the non-foreground edge pixel area in the depth map sub-area, which can reduce the compression loss of the depth map stitched image and thereby improve the reconstruction quality of the reconstructed free viewpoint video.

[0096] Furthermore, the pixel points of the depth map sub-area corresponding to each viewpoint contained in the depth map stitching image are part of the pixel points in a set of pixel points in the texture map stitching image that correspond one-to-one to the pixel points in the texture map sub-area of ​​the corresponding viewpoint, which can reduce the resolution of the depth map stitching image, thereby further saving transmission resources.

[0097] In another video processing method of an embodiment of the present specification, by determining the encoding category to which the encoding method belongs, when it is determined that the encoding method belongs to custom depth map encoding, the texture maps of multiple viewpoints of frame synchronization are spliced ​​into a texture map spliced ​​image, the depth maps corresponding to the texture maps of the multiple viewpoints of frame synchronization are spliced ​​into a depth map spliced ​​image, and the texture map spliced ​​image and the depth map spliced ​​image are respectively encoded to obtain a corresponding texture map compressed image and a depth map compressed image; the encoding category identifier and encoding identifier information corresponding to the encoding method are stored in the image set user information. Through this video processing process, the texture map and the depth map can be spliced ​​separately and encoded and compressed separately, so that the corresponding encoding method can be adopted based on the different image features of the texture map and the depth map, thereby reducing compression loss and reducing the requirements for the decoding performance of the decoding end.

[0098] Furthermore, by storing the scanning mode identification information corresponding to the scanning mode in the scanning mode information area in the video compression frame header, there is no need to store specific stitching rules and viewpoint identifications of the texture map stitching image and the depth map stitching image in the video compression frame header, thereby saving transmission resources.

[0099] In another aspect of the embodiments of the present specification, by decapsulating the free viewpoint video stream, a video compression frame and the encoding category information of the encoding method of the video compression frame are obtained. Based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts custom depth map encoding, the texture map compressed image and the depth map compressed image in the video compression frame are decoded respectively to obtain a texture map spliced ​​image and a depth map spliced ​​image; wherein the texture map spliced ​​image includes texture maps of multiple synchronized viewpoints; and the depth map spliced ​​image includes depth maps corresponding to the texture maps of the multiple viewpoints. It can be seen from this video processing process that when it is determined that the video compression frame adopts custom depth map encoding, the texture map compressed image and the depth map compressed image in the video compression frame can be decoded respectively, so that different decoding resources can be used to decode the texture map compressed image and the depth map compressed image respectively, so that the resources of the decoding end can be fully utilized to avoid being limited by the decoding performance and decoding capability of one decoding resource.

[0100] In addition, multi-view joint coding can remove redundant information between viewpoints and improve coding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 This is a schematic diagram of a specific application system for free viewpoint video display in an embodiment of this specification;

[0102] Figure 2 This is a schematic diagram of an interactive interface of a terminal device in an embodiment of this specification;

[0103] Figure 3 It is a schematic diagram of a collection device setting method in an embodiment of this specification;

[0104] Figure 4 This is another schematic diagram of the interactive interface of a terminal device in an embodiment of this specification;

[0105] Figure 5 is a schematic diagram of a free viewpoint video data generation process in an embodiment of this specification;

[0106] Figure 6 is a schematic diagram of generation and processing of 6DoF video data in an embodiment of this specification;

[0107] Figure 7 It is a schematic diagram of the structure of a data header file in an embodiment of this specification;

[0108] Figure 8 is a schematic diagram of 6DoF video data processing on the user side in an embodiment of this specification;

[0109] Fig. 9 is a structural schematic diagram of a spliced ​​image of a video frame in an embodiment of this specification;

[0110] Fig.10 is a flow chart of a video processing method in an embodiment of this specification;

[0111] Fig.11 It is a schematic diagram of a texture image stitching process in a specific application scenario in an embodiment of this specification;

[0112] Fig.12 is a schematic diagram of a depth map stitching process in a specific application scenario in an embodiment of this specification;

[0113] Fig.13 It is a schematic diagram of a raster scanning method in the embodiment of this specification;

[0114] Fig.14 It is a schematic diagram of a method for setting a viewpoint identifier corresponding to a raster scanning method in an embodiment of this specification;

[0115] Fig.15 is a flow chart of another video processing method in an embodiment of this specification;

[0116] Fig.16 This is a schematic diagram of a process of encoding using custom depth map encoding in an embodiment of this specification;

[0117] Fig.17 is a schematic diagram of a process of encoding using multi-view joint encoding in an embodiment of this specification;

[0118] Fig.18 is a schematic diagram of the structure of a multimedia channel in an embodiment of this specification;

[0119] Fig.19 This is a schematic diagram of specific types and regional distribution of user information of an image set in an embodiment of this specification;

[0120] Fig. 20 It is a schematic diagram of the format of a video compression frame in a video channel in an embodiment of this specification;

[0121] Fig.21 is a flow chart of a video processing method in an embodiment of this specification;

[0122] Fig. 22 is a flowchart of a decoding method in an embodiment of this specification;

[0123] Fig.23 is a flowchart of another decoding method in an embodiment of this specification;

[0124] Fig.24 is a flow chart of another video processing method in an embodiment of this specification;

[0125] Fig.25 is a flow chart of another video processing method in an embodiment of this specification;

[0126] Fig.26 is a flow chart of another video processing method in an embodiment of this specification;

[0127] Fig. 27 is a structural schematic diagram of a video processing device in an embodiment of this specification;

[0128] Fig.28 is a structural schematic diagram of another video processing device in an embodiment of this specification;

[0129] Fig.29 is a structural schematic diagram of another video processing device in an embodiment of this specification;

[0130] Fig.30 is a structural schematic diagram of another video processing device in an embodiment of this specification;

[0131] Fig.31 It is a structural schematic diagram of an electronic device in an embodiment of this specification;

[0132] Fig.32 is a schematic diagram of the structure of another electronic device in the embodiments of this specification;

[0133] Fig.33 is a schematic diagram of the structure of another electronic device in the embodiments of this specification;

[0134] Fig.34 It is a structural diagram of a video processing system in an embodiment of this specification. DETAILED DESCRIPTION

[0135] In order to enable those skilled in the art to better understand and implement the embodiments in this specification, the following first provides an exemplary introduction to the implementation of free viewpoint video in conjunction with the accompanying drawings and specific application scenarios.

[0136] refer to Figure 1 In the embodiment of the present invention, a specific application system for free viewpoint video display may include a collection system 11, a server 12 and a display device 13 of multiple collection devices, wherein the collection system 11 may collect images of the viewing area; the collection system 11 or the server 12 may process the acquired synchronized multiple texture maps to generate multi-angle free viewpoint data that can support the display device 13 to switch virtual viewpoints. The display device 13 may display a reconstructed image generated based on the multi-angle free viewpoint data, the reconstructed image corresponds to a virtual viewpoint, and may display reconstructed images corresponding to different virtual viewpoints according to user instructions, switching the viewing position and viewing angle.

[0137] In a specific implementation, the process of reconstructing an image and obtaining a reconstructed image can be implemented by the display device 13, or by a device located in a content delivery network (Content Delivery Network, CDN) in an edge computing manner. It is understandable that Figure 1 This is only an example and is not intended to limit the collection system, server, terminal device, or specific implementation method.

[0138] Continue to refer Figure 1 , the user can view the area to be viewed through the display device 13. In this embodiment, the area to be viewed is a basketball court. As mentioned above, the viewing position and viewing angle can be switched.

[0139] For example, the user can slide on the screen to switch the virtual viewpoint. Figure 2 , the user's finger moves along D 22 When you slide the screen in the right direction, you can switch the virtual viewpoint for viewing. Figure 3 , the position of the virtual viewpoint before sliding can be VP 1 , after sliding the screen to switch the virtual viewpoint, the position of the virtual viewpoint can be VP 2 . Combined with reference Figure 4 After sliding the screen, the reconstructed image displayed on the screen can be Figure 4The reconstructed image can be obtained by reconstructing the image based on multi-angle free view data generated by images collected by multiple collection devices in an actual collection scenario.

[0140] It is understandable that the image viewed before switching may also be a reconstructed image. The reconstructed image may be a frame image in a video stream. In addition, the method of switching the virtual viewpoint according to the user's instruction may be various, which is not limited here.

[0141] In a specific implementation, the viewpoint can be represented by 6 degrees of freedom (DoF) coordinates, where the spatial position of the viewpoint can be represented as (x, y, z) and the viewing angle can be represented as three rotation directions: Accordingly, based on the 6-DOF coordinates, the virtual viewpoint, including the position and viewing angle, can be determined.

[0142] Virtual viewpoint is a three-dimensional concept, and three-dimensional information is required to generate a reconstructed image. In a specific implementation, the multi-angle free viewpoint data may include depth map data to provide the third dimension information outside the plane image. Compared with other implementations, such as providing three-dimensional information through point cloud data, the amount of depth map data is relatively small.

[0143] In the embodiment of the present invention, the switching of the virtual viewpoint can be performed within a certain range, which is the multi-angle free viewing angle range. That is, within the multi-angle free viewing angle range, the position and viewing angle of the virtual viewpoint can be switched arbitrarily.

[0144] The multi-angle free viewing angle range is related to the arrangement of the acquisition equipment. The wider the shooting coverage of the acquisition equipment, the larger the multi-angle free viewing angle range. The image quality displayed by the terminal device is related to the number of acquisition devices. Generally, the more acquisition devices are set, the fewer the hollow areas in the displayed image.

[0145] In addition, the range of the multi-angle free viewing angle is related to the spatial distribution of the acquisition device. The range of the multi-angle free viewing angle and the interaction mode with the display device on the terminal side can be set based on the spatial distribution relationship of the acquisition device.

[0146] Those skilled in the art will appreciate that the above embodiments and corresponding drawings are for illustrative purposes only and are not limitations on the settings of acquisition devices and the relationship between multi-angle free viewing angles, nor are they limitations on the interaction methods and display effects of display devices.

[0147] Combined with reference Figure 5In order to reconstruct free viewpoint video, texture image acquisition and depth map calculation are required, which includes three main steps: multi-camera video capture, camera parameter estimation, and depth map calculation. For multi-camera video capture, it is required that the videos captured by each camera can be aligned at the frame level. Among them, the texture image can be obtained through multi-camera video acquisition; the camera parameters can be obtained through camera parameter estimation, which can include camera internal parameter data and external parameter data; the depth map can be obtained through depth map calculation. Multiple synchronized texture images and depth maps of corresponding viewing angles and camera parameters form 6DoF video data.

[0148] In the embodiments of this specification, no special camera, such as a light field camera, is required to capture the video. Similarly, no complicated camera calibration is required before the capture. The positions of multiple cameras can be laid out and arranged to better capture the objects or scenes that need to be captured.

[0149] After the above three steps are completed, the texture map collected from multiple cameras, the camera parameters of all cameras, and the depth map of each camera are obtained. These three parts of data can be called data files in multi-angle free-view video data, or 6-degree-of-freedom video data (6DoF video data). With this data, the user end can generate virtual viewpoints based on the virtual 6-degree-of-freedom (DoF) position, thereby providing a 6DoF video experience.

[0150] Combined with reference Figure 6 , 6DoF video data and indicative data can be compressed and transmitted to the user side, and the user side can obtain the user side 6DoF expression, that is, the aforementioned 6DoF video data and metadata, based on the received data. Among them, the indicative data can also be called metadata (Metadata), wherein the video data includes texture maps and depth map data of each viewpoint corresponding to multiple cameras, and the texture map and the depth map can be spliced ​​according to certain splicing rules or splicing patterns to form a spliced ​​image.

[0151] Combined with reference Figure 7, metadata can be used to describe the data mode of 6DoF video data, which can include: stitching pattern metadata, which is used to indicate the storage rules of pixel data of multiple texture maps and depth map data in the stitched image; padding pattern metadata, which can be used to indicate the way to perform edge protection in the stitched image, and other metadata. Metadata can be stored in the data header file, and the specific storage order can be as follows: Figure 7 as shown, or stored in another order.

[0152] Combined with reference Figure 8 The user side obtains 6DoF video data, which includes camera parameters, stitched images (texture maps and depth maps), and metadata (metadata), in addition to the user's interactive behavior data. With this data, the user side can use 6DoF rendering based on depth image rendering (DIBR, Depth Image-Based Rendering) to generate a virtual viewpoint image at a specific 6DoF position generated according to user behavior, that is, according to the user's instructions, determine the virtual viewpoint at the 6DoF position corresponding to the instruction.

[0153] At present, the texture maps and corresponding depth maps of multiple synchronized viewpoints are usually spliced ​​to obtain a spliced ​​image, and the spliced ​​image is uniformly encoded and transmitted. Fig. 9 The structural schematic diagram of the stitched image shown in FIG. 1 stitches together the texture map of viewpoint 1 to viewpoint 8 and the depth map of viewpoint 1 to viewpoint 8 to form a stitched image 90, and then the stitched image 90 can be encoded using an encoding method to obtain a video compression frame, and then the video compression frame can be transmitted.

[0154] The inventors have found through research that although the above video processing method has strong universal application, due to the large differences in the characteristics of the texture map and the depth map, it is difficult to use differentiated encoding tools and strategies to process the spliced ​​image containing the texture map and the depth map, which will cause compression loss to the spliced ​​image, thereby affecting the image quality of the reconstructed free viewpoint video. For example, the resolution of the spliced ​​image obtained after splicing is relatively large, and the decoding capability of the decoding end may be difficult to meet the requirements.

[0155] In summary, the existing video processing methods have many limitations.

[0156] In response to the above problems, an embodiment of the present specification provides a video processing method, wherein, based on the different image characteristics of the texture map and the depth map, the texture maps and the corresponding depth maps of multiple synchronized viewpoints are spliced ​​respectively to obtain a texture map spliced ​​image and a depth map spliced ​​image, and the texture map spliced ​​image and the depth map spliced ​​image are encoded respectively, so as to obtain the corresponding texture map compressed image and depth map compressed image.

[0157] Since the texture map stitching image and the depth map stitching image are encoded separately, the corresponding encoding method can be used for encoding compression based on the characteristics of the texture map and the depth map, thereby reducing the compression loss and further improving the image quality of the free viewpoint video reconstructed based on the texture map stitching image and the depth map stitching image.

[0158] In order to enable those skilled in the art to better understand and implement the embodiments of the present specification, the video processing process from both the encoding side and the decoding side is described in detail below with reference to the accompanying drawings.

[0159] First, the video processing method on the encoding side is explained in detail.

[0160] Reference Fig.10 The flowchart of the video processing method shown in the embodiment of this specification may specifically include the following steps:

[0161] S101, stitching texture images of multiple viewpoints with frame synchronization into a texture image stitching image.

[0162] In a specific implementation, the texture images of the multiple viewpoints of the frame synchronization can be scanned according to a preset scanning method, and the texture image splicing image can be obtained by splicing according to a preset splicing rule, such as Fig.11 As shown in the schematic diagram of the texture map stitching process, for the four video streams synchronously captured by camera 1 to camera 4, the texture map stitching image video stream can be obtained according to the frame timing.

[0163] S102, stitching the depth maps corresponding to the texture maps of the multiple viewpoints of the frame synchronization into a depth map stitching image.

[0164] Similar to step S101, the depth maps corresponding to the texture maps of the multiple viewpoints of the frame synchronization may be scanned according to the preset scanning method, and the depth map stitching image may be obtained by stitching them according to the preset stitching rule. Fig.12 As shown in the schematic diagram of the depth map stitching process, for the depth map video streams corresponding to cameras 1 to 4, the depth map stitching image video streams can be obtained according to the frame timing.

[0165] In a specific implementation, the specific scanning method can be raster scanning (RasterScan), zigzag scanning (ZigZag-Scan) and many other methods.

[0166] Raster scanning means scanning from left to right and from top to bottom, first scanning one row and then moving to the starting position of the next row to continue scanning. Fig.13 The schematic diagram of the raster scanning method is shown in FIG. In the Z-shaped scanning, Z is a figurative representation. In a specific embodiment, the texture map and the depth map of each viewpoint in the space can be acquired and spliced ​​in sequence according to the scanning trajectory.

[0167] It is understandable that the above scanning methods are only examples, and the embodiments of this specification do not limit the specific scanning methods.

[0168] In some embodiments of this specification, a fixed scanning method is preset for scanning. For example, the encoding end device and the decoding end device may agree to use a raster scanning method for scanning, so that during transmission, only the identification information corresponding to the texture map and the depth map of each viewpoint can be transmitted without transmitting the identification information corresponding to the scanning method, and the decoding end device can decode according to the same scanning order.

[0169] In the embodiments of the present specification, for viewpoints arranged in any manner in space, in order to simplify the viewpoint information storage rules in the corresponding texture map sub-regions and depth map sub-regions in the texture map stitching image and the depth map stitching image, the setting rules of the viewpoint identification can be pre-set.

[0170] In some embodiments of this specification, Fig.14 The schematic diagram of the viewpoint identification setting method shown in FIG. 1 is a schematic diagram of the viewpoint identification setting method, and each camera is used as the original viewpoint. First, a main camera can be selected to ensure synchronous acquisition, and the other cameras are slave cameras. The main camera is marked as 0. The selection rule of the main camera is not limited in the embodiments of this specification. After that, a spatial coordinate system can be established with the main camera as the origin, such as Fig.14 As shown, after that, the data can be sorted according to the spatial coordinate system (x, y, z). An example sorting rule is as follows:

[0171] a. Sort by the priority of the small numbers of the z coordinates. The smaller the number of the camera, the smaller the number;

[0172] b. Sort by the order of the small numbers of the y coordinates based on the z coordinate sorting. The smaller the number, the smaller the camera number.

[0173] c. Sort by the order of the x-coordinate numbers based on the y-coordinate order. The smaller the number, the smaller the camera number.

[0174] An example of viewpoint sorting obtained according to the above viewpoint sorting rules is as follows:

[0175] according to Fig.14 , and select any camera as the main camera and set its number to 0. Then, the coordinates of the 7 slave cameras in the coordinate system are obtained (-1,1,1),(-1,0,1),(1,0,1),(1,0,0),(-1,0,-1),(0,0,-1),(1,0,-1). According to the above sorting rules, they are sorted to obtain (-1,0,-1),(0,0,-1),(1,0,-1),(1,0,0),(-1,0,1),(1,0,1),(-1,1,1).

[0176] The final camera numbers are as follows:

[0177] (0,0,0) camera number 0;

[0178] (-1,0,-1) camera number 1;

[0179] (0,0,-1) camera number 2;

[0180] (1,0,-1) camera number 3;

[0181] (1,0,0) camera number 4;

[0182] (-1,0,1) camera number 5;

[0183] (1,0,1) camera number 6;

[0184] (-1,1,1) Camera number 7.

[0185] The order of the camera labels is determined by the above process, that is, the raster scanning order is determined. The above camera identification can also represent the order of obtaining the texture map and depth map of each viewpoint. Therefore, the texture map of the multiple viewpoints synchronized with the frame can be scanned in a raster scanning manner, and the texture maps of the multiple viewpoints can be spliced ​​in a raster scanning order to obtain the texture map spliced ​​image; and the depth map corresponding to the texture map of the multiple viewpoints synchronized with the frame can be scanned in the raster scanning manner, and the texture maps of the multiple viewpoints can be spliced ​​in a raster scanning order to obtain the depth map spliced ​​image.

[0186] S103, respectively encoding the texture map stitching image and the depth map stitching image to obtain a corresponding texture map compressed image and depth map compressed image.

[0187] In a specific implementation, according to different features of the texture map stitching image and the depth map stitching image, the texture map stitching image and the depth map stitching image may be encoded using different encoding methods respectively.

[0188] For example, the depth map stitched image may be encoded using a coding method based on a region of interest to obtain a corresponding depth map compressed image. The depth map stitched image is encoded using a coding method based on a region of interest, so that compression encoding can be performed according to the image features of the depth map stitched image, thereby reducing the compression loss of the depth map, and further improving the image quality of the free viewpoint video reconstructed based on the depth map.

[0189] As a specific example, a region of interest (ROI) pixel area in the depth map stitched image is encoded based on a preset constant quantization parameter to obtain a corresponding depth map compressed image.

[0190] The inventor has found through research that the quality of the foreground edge pixel area in the depth map sub-area is very critical to the image quality reconstructed based on the virtual viewpoint. Based on this, in order to improve the image quality of the reconstructed virtual viewpoint, in a specific implementation of this specification, the foreground edge pixel area in the depth map sub-area corresponding to each viewpoint contained in the depth map stitched image is encoded using a first constant quantization parameter, and the non-foreground edge pixel area in the depth map sub-area corresponding to each viewpoint is encoded using a second constant quantization parameter, and the parameter value of the first constant quantization parameter is less than the parameter value of the second constant quantization parameter.

[0191] In a specific implementation, for a texture map stitching image, as needed, one can be selected from a variety of coding methods suitable for texture maps, such as VP8, VP9, ​​AV1, H.264 / AVC, H.265 / HEVC, Versatile Video Coding (VVC), Audio and Video Coding Standard (AVS), AVS+, AVS2, AVS3, etc., to encode the texture map stitching image and obtain a texture map compressed image.

[0192] Among them, VP8 and VP9 are open video compression coding methods. H.264 / AVC was jointly developed by ITU-T and ISO / IEC, and is positioned to cover the entire video application field. ITU-T named this standard H.264 (formerly called H.26L), and ISO / IEC called it MPEG-4 Advanced Video Coding (AVC). H.265 is a new video coding standard developed by ITU-T VCEG after H.264. The full name of the standard is High Efficiency Video Coding (HEVC). AVS+, AVS2 and AVS3 are the optimization and evolution technologies of AVS.

[0193] In a specific implementation, step S101 and step S102 may be executed sequentially (step S101 may be executed first, or step S102 may be executed first), or may be executed in parallel.

[0194] As an optional step, after step S103, the following steps may be performed:

[0195] S104: Encapsulate the texture map compressed image and the depth map compressed image in the same video channel.

[0196] By encapsulating the texture map compressed image and the depth map compressed image in the same video channel, that is, the texture map compressed image and the depth map compressed image can reuse one video channel, so that the hardware interface of the existing hardware device can be reused, and the specifications of the existing open source framework are complied with, and the depth map and texture map are also easily synchronized.

[0197] In a specific implementation, the depth map compressed image may be encapsulated in a frame header region of the video compression frame.

[0198] The depth map compressed image is encapsulated in the frame header area of ​​the video compression frame. When different decoding resources are used at the decoding end to decode the texture map compressed image and the depth map compressed image respectively, the reading of irrelevant data in the depth map compressed image decoding process can be reduced, thereby improving data processing efficiency and saving processing resources.

[0199] In a specific implementation, the pixel points of the depth map sub-region corresponding to each viewpoint contained in the depth map stitched image are all or part of the pixel points in the pixel point set in the texture map stitched image that corresponds one-to-one with the pixel points in the texture map sub-region of the corresponding viewpoint. In some embodiments of the present specification, the depth map stitched image is obtained by downsampling the pixels in the original depth map or the original depth map stitched image, and the pixel points of the depth map sub-region corresponding to each viewpoint contained in the depth map stitched image obtained by downsampling are part of the pixel points in the pixel point set in the texture map stitched image that corresponds one-to-one with the pixel points in the texture map sub-region of the corresponding viewpoint, so that the resolution of the depth map stitched image can be reduced, thereby further saving transmission resources.

[0200] In specific implementation, the above embodiments may be further expanded. The following is an illustration through some specific embodiments.

[0201] In order to expand the compatibility of compression coding applicable to the embodiments of this specification, multiple coding methods can be set, including custom coding of depth maps. In some embodiments of this specification, reference is made to Fig.15 The flowchart of the video processing method shown in the figure can be implemented by the following steps:

[0202] S151, determine the encoding category to which the encoding method belongs. If it is determined that the encoding method belongs to depth map custom encoding, step S152 can be executed; if it is determined that the encoding method belongs to multi-view joint encoding, step S153 can be executed.

[0203] S152, storing the encoding identification information corresponding to the encoding method into the encoding information area in the image set user information; and stitching the texture maps of multiple viewpoints synchronized by frames into a texture map stitching image, stitching the depth maps corresponding to the texture maps of multiple viewpoints synchronized by frames into a depth map stitching image, and encoding the texture map stitching image and the depth map stitching image respectively to obtain the corresponding texture map compressed image and depth map compressed image.

[0204] In a specific implementation, when it is determined that the encoding method belongs to custom encoding of the depth map, the encoding system or the electronic device used for encoding can perform two actions: one is to store the encoding identification information corresponding to the encoding method in the encoding information area in the image set user information, so that the decoding system or the electronic device used for decoding can recognize and use the corresponding decoding method for decoding.

[0205] Another aspect is the execution of encoding using the depth map custom encoding. Fig.16As shown, it is a schematic diagram of the encoding process using custom encoding of depth maps, wherein, for the texture maps T1~T4 of the four viewpoints of frame synchronization, they can be spliced ​​according to a preset scanning method to obtain a texture map splicing image Tx, and then the texture map splicing image Tx is encoded using a first encoding method to obtain a texture map compressed image Te; for the depth maps D1~D4 corresponding to the texture maps T1~T4 of the four viewpoints of frame synchronization, they can be spliced ​​according to a preset scanning method to obtain a depth map splicing image Dx, and then the depth map splicing image Dx is encoded using a second encoding method to obtain a depth map compressed image De. In a specific implementation, different encoding methods are used for the texture map splicing image Tx and the depth map splicing image Dx, so that a more matching encoding method can be used for encoding based on the different characteristics of the texture map and the depth map.

[0206] In some embodiments of the present specification, the texture maps T1~T4 of the multiple viewpoints synchronized with the frame can be scanned in a raster scanning manner, and the texture maps of the multiple viewpoints can be spliced ​​in a raster scanning order to obtain the texture map spliced ​​image Tx; and the depth maps D1~D4 corresponding to the texture maps of the multiple viewpoints synchronized with the frame can be scanned in the raster scanning manner, and the texture maps of the multiple viewpoints can be spliced ​​in a raster scanning order to obtain the depth map spliced ​​image Dx.

[0207] In some embodiments of the present specification, the first encoding method can be selected from a variety of encoding methods suitable for texture maps, such as VP8, VP9, ​​AV1, H.264 / AVC, H.265 / HEVC, VVC, AVS, AVS+, AVS2, AVS3, etc.; the second encoding method can be encoded in a region of interest-based encoding method. For example, for the region of interest (Region Of Internet, ROI) pixel area in the depth map stitched image, it can be encoded based on a preset constant quantization parameter to obtain a corresponding depth map compressed image.

[0208] In a specific implementation, an encoder or encoding software that supports a corresponding encoding method may be used for implementation.

[0209] S153, obtaining the viewpoint identifier and frame identifier of the texture map and depth map of the multiple viewpoints synchronized with the frame, and storing them in the image information area in the image set user information, and storing the encoding identifier information corresponding to the encoding method in the encoding information area in the image set user information; and encoding the texture map and depth map of the multiple viewpoints synchronized with the frame respectively using a preset multi-view joint encoding method to obtain a joint compressed image.

[0210] In some embodiments of the present specification, in addition to custom encoding of depth maps, the encoding categories also include encoding types such as multi-view joint encoding. Therefore, according to needs, in addition to custom encoding of depth maps, multi-view joint encoding can also be used to encode texture maps and depth maps of multiple viewpoints.

[0211] By adopting multi-view joint coding, redundant information between viewpoints can be removed and coding efficiency can be improved.

[0212] In a specific implementation, when it is determined that the encoding method belongs to multi-view joint encoding, the encoding system or the electronic device used for encoding can perform two actions: one is to obtain the viewpoint identifier and frame identifier of the texture map and depth map of the multiple viewpoints of the frame synchronization, and store them in the image information area in the image set user information, and store the encoding identification information corresponding to the encoding method in the encoding information area in the image set user information, so that the decoding system or the electronic device used for decoding can recognize and use the corresponding decoding method for decoding. On the other hand, the execution action of encoding using the multi-view joint encoding method.

[0213] like Fig.17 As shown in FIG. 1 , a schematic diagram of a process of encoding using multi-view joint encoding in an embodiment of the present specification is shown. For texture maps and depth maps of multiple viewpoints (viewpoint 0 to viewpoint N-1) with frame synchronization, first, a viewpoint is selected as a texture map of an independent viewpoint as an independent view, such as Fig.17 As shown, the texture map of viewpoint 0 is selected as an independent view and encoded by a video encoding device compliant with HEVC. On the one hand, the texture map compressed image obtained by encoding is output to an encapsulation device. On the other hand, a preset depth map encoding device is used to encode the depth map of viewpoint 0 with reference to the information of the texture compressed image output by the video encoding device compliant with HEVC. The depth map compressed image obtained after encoding is output to the encapsulation device for further encapsulation on the one hand, and is output to video encoding devices for non-independent views and depth map encoding devices for non-independent views of other viewpoints on the other hand, so as to be used as a reference when the video encoding device for non-independent views encodes the texture map of the corresponding viewpoint, and to be used as a reference when the depth map encoding device for non-independent views encodes the depth map of the corresponding viewpoint, thereby obtaining a texture map compressed image and a depth map compressed image obtained by jointly encoding multiple encoding devices such as a video encoding device compliant with HEVC, a depth map encoding device, and a video encoding device for non-independent views and a depth map encoding device for non-independent views.

[0214] In a specific implementation, each encoding device may adopt the same encoding method, or may select different encoding methods according to different characteristics of the texture map and the depth map.

[0215] In a specific implementation, the specific multi-view joint encoding method can be selected from encoding methods such as VP8, VP9, ​​AV1, H.264 / AVC, H.265 / HEVC, VVC, AVS, AVS+, AVS2, and AVS3.

[0216] In a specific implementation, in order to facilitate decoding at the decoding end, the indication information related to the encoding, such as the encoding category identifier corresponding to the encoding method, the encoding identification information of the specific encoding method, etc., can be stored in the image set user information.

[0217] Through the aforementioned step S152 or step S153, the coding compression of the video image can be realized. For the purpose of video transmission and synchronous playback of different multimedia contents, the multimedia contents (video, audio, subtitles, chapter information, etc.) generated by the encoder can be packaged. The specific packaging format can be understood as a container of the media. Through packaging, the coding data of the video channel and the coding data of the audio channel can be packaged into a complete multimedia file.

[0218] In a specific implementation, based on different encoding methods, specific encoded video frames, encoding information, encapsulation information, etc. may all be different. In the embodiments of this specification, encoding information, encapsulation information, etc. may all be regarded as image set user information. Based on different encapsulation formats and encoding methods, different types of user information in the image set user information may be stored in one area, or may be set in different storage areas as needed. This will be explained below through specific application scenarios.

[0219] The following first describes the packaging methods of texture map compressed images and depth map compressed images encoded by the depth map custom encoding method and the multi-view joint encoding method, and then introduces some examples of storage methods of image set user information in combination with the depth map custom encoding method and the multi-view joint encoding method.

[0220] In the embodiment of the present specification, the texture map compressed image and the depth map compressed image encoded in step S152 or step S153 may reuse one video channel. Specifically, the texture map compressed image and the depth map compressed image may be encapsulated in a video compression frame of the same video channel, such as Fig.18 The multimedia information channel structure diagram shown in FIG. 180 includes a video channel 181 and an audio channel 182. To transmit the encoded texture map compressed image and the depth map compressed image, the texture map compressed image and the depth map compressed image can be encapsulated in the video channel 181. More specifically, the texture map compressed image and the depth map compressed image can be encapsulated in the video compression frame of the same video channel. Fig.18In the video channel 181, a series of video compression frames are encapsulated in time sequence, and each video compression frame 18F includes a frame header 18A and a frame body 18B.

[0221] In some embodiments of the present specification, the depth map compressed image obtained by using the depth map encoding method can be encapsulated into the depth map area in the frame header 18A of the video compression frame 18F of the video channel 181, and the texture map compressed image can be encapsulated in the frame body 18B of the video compression frame 18F of the video channel 181. In some embodiments of the present specification, the texture map compressed image and the texture map compressed image obtained by encoding in step S153 can also be encapsulated in the video compression frame of the video channel. Specifically, the texture map compressed image of each viewpoint synchronized and the depth map compressed image of the same viewpoint can be stored in the frame body 18B of the video compression frame 18F in sequence.

[0222] The following uses examples of depth map custom encoding and multi-view joint encoding to illustrate the specific storage method of the user information of the image set after encapsulation.

[0223] In a specific implementation, the coding category identifier corresponding to the coding method can be stored in the image set user information area in the channel header of the video channel. The coding category identifier is a subset of the image set user information. In order to achieve fast decoding and save storage resources, it can be stored in the image set user information area in the channel header of the video channel. For example, if there are only two coding categories, namely, depth map custom coding and multi-view joint coding, a 1-bit character can be specially set in the channel header information to identify whether the video channel contains a video compression frame obtained by the depth map custom coding method or a video compression frame obtained by the multi-view joint coding method. If there are more categories of coding methods, more bits can be reserved for identifying the coding category to which it belongs. At the same time, the encoding identification information can be stored in the encoding information area in the frame header of the video compression frame. For example, if multi-view joint encoding is used, the specific identification information in the frame header of the video compression frame can identify the specific encoding method for the texture map and the depth map, whether it is 3D-HEVC, 3D-AVC, etc.; if custom encoding of the depth map is used, the specific identification information in the frame header of the video compression frame can identify the specific encoding method for the texture map spliced ​​image and the depth map spliced ​​image, whether it is AVC, HEVC, AVS, AVS2, AVS3, etc.

[0224] In addition, corresponding to the joint compressed image obtained by the multi-view joint encoding method, in a specific implementation, the viewpoint identifier contained in the joint compressed image and the frame identifier corresponding to the joint stitched image can be stored in the image information storage area in the video compression frame header.

[0225] Reference Fig.19As shown in the schematic diagram of specific types and regional distribution of user information of the image set, in a specific implementation, a coding category area Ca1 can be set in the channel header Ca of the video channel to store a coding category identifier to characterize the coding category to which a specific coding method belongs, such as custom coding of the depth map, or multi-view joint coding, and the coding identification information is stored in the coding information area Ia1 in the frame header Ia of the video compression frame I to characterize the specific coding method used in the custom coding of the depth map or the multi-view joint coding, such as which one or more coding methods are 3D-HEVC, 3D-AVC, AVC, HEVC, AVS, etc.

[0226] In a specific implementation, as the extended information in the image set user information, continue to refer to Fig.19 Corresponding to the multi-view joint coding method, the frame header Ia of the video compression frame I may include the image information storage area Ia2, which can store the viewpoint identifier and frame identifier of the texture map and depth map targeted by the multi-view joint coding.

[0227] As described in the previous embodiment, corresponding to multi-view joint coding, the texture map compressed image and the depth map compressed image of each viewpoint can be arranged in sequence in an interlaced manner in the frame body Ib, such as Fig. 20 : A schematic diagram of the format of a video compression frame in a video channel is shown, wherein the texture map compressed image and the depth map compressed image of each viewpoint obtained in the video compression frame I can be arranged adjacent to each other as an image combination, and the images of each viewpoint are arranged in sequence according to the processing order of the encoding device, and the viewpoint identifier and the frame identifier of each texture map compressed image and the depth map compressed image are arranged in the same order in the image information storage area Ia2. For example, for the same frame, the viewpoint identifiers corresponding to the texture map compressed image and the depth map compressed image can be arranged as an array, for example, it can be expressed as V0D0V1D1V2D2..., wherein V represents the texture map and D represents the depth map.

[0228] Corresponding to the custom encoding of the depth map, only the texture map compressed image is stored in the frame body Ib, and the depth map compressed image can be stored in the depth map storage area Ia3 of the frame header Ia.

[0229] Understandably, Fig.19 and Fig. 20 Only a schematic structure of a video compression frame in a video channel is illustrated. In a specific implementation, multiple video compression frames are sequentially stored in the video channel to form a video stream.

[0230] After the above processing, the free viewpoint video stream can be transmitted through the video channel, and then the free viewpoint video can be played based on the free viewpoint video stream.

[0231] In order to enable those skilled in the art to better understand the principles and specific implementation schemes of the methods for processing videos through encoding and encapsulation in the embodiments of this specification, the following corresponding explanations are given on the decapsulation and decoding of the acquired free viewpoint video stream after transmission to the decoding end such as the terminal side, as well as the reconstruction of the free viewpoint video.

[0232] Reference Fig.21 As shown in the flowchart of the video processing method, at the free viewpoint video stream receiving side, such as a decoding end device, the obtained free viewpoint video stream can be processed by the following steps:

[0233] S211, decapsulating the free viewpoint video stream to obtain video compression frames and encoding category information of the encoding method of the video compression frames.

[0234] In a specific implementation, corresponding to the encapsulation method in the aforementioned embodiment, the channel header of the video channel is decapsulated to obtain the encoding category information to which the encoding method of the video compression frame belongs.

[0235] S212, based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts the depth map custom encoding, respectively decode the texture map compressed image and the depth map compressed image in the video compression frame to obtain the texture map stitching image and the depth map stitching image.

[0236] The texture map stitching image includes texture maps of multiple synchronized viewpoints; and the depth map stitching image includes depth maps corresponding to the texture maps of the multiple viewpoints.

[0237] S213, based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts multi-view joint encoding, the texture map compressed image and the depth map compressed image in the video compression frame are decoded using the corresponding multi-view joint decoding method to obtain synchronized texture maps of multiple viewpoints and corresponding depth maps.

[0238] Step S213 is an optional step, which can realize decoding of the video compression frame obtained by multi-view joint coding.

[0239] For step S212 or step S213, refer to Fig. 22 The flowchart of the decoding method shown in the figure shows that for a video compression frame with a custom depth map encoding method, the following decoding method can be used for decoding:

[0240] S2121, decode the frame header of the video compression frame, and obtain the encoding identification information of the texture map compression image and the encoding identification information of the depth map compression image from the encoding information storage area in the frame header.

[0241] S2122: Decode the depth map compressed image using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain the depth map stitched image.

[0242] S2123: Decode the texture map compressed image using a decoding method corresponding to the coding identification information of the texture map compressed image to obtain the texture map spliced ​​image.

[0243] In a specific implementation, different decoding resource distributions may be used for decoding in step S2122 and step S2123. For a video compression frame obtained by using a depth map encoding method, as in the example of the previous embodiment, the depth map compressed image is stored in the depth map storage area of ​​the frame header, and the texture map compressed image is stored in the frame body Ib, such as Fig.19 As shown, the texture map compressed image can be decoded using the first decoding resource to obtain a texture map spliced ​​image, and the depth map compressed image can be decoded using the second decoding resource to obtain a depth map spliced ​​image.

[0244] Reference Fig.23 As shown in the flowchart of another decoding method, for the multi-view joint coding method, the decoding can be performed using the following steps:

[0245] S2131, decode the frame header of the video compression frame, and obtain the coding identification information of the texture map compression image and the coding identification of the depth map compression image from the coding information storage area in the frame header.

[0246] S2132, decoding the texture map compressed image and the depth map compressed image using a multi-view joint decoding method corresponding to the coding identification information of the texture map compressed image and the coding identification of the depth map compressed image, to obtain the texture maps of the synchronized multiple viewpoints and the corresponding depth maps.

[0247] For the video compression frame obtained by multi-view joint coding, the texture map compressed image and the depth map compressed image are interleaved and stored in the frame body Ib, such as Fig. 20 As shown. The texture map compressed image can be decoded using the first decoding resource to obtain a texture map, and the depth map compressed image can be decoded using the second decoding resource to obtain a depth map. In addition to the texture map of the independent viewpoint, the texture map and depth map of other viewpoints can refer to the texture map or depth map obtained by decoding other viewpoints during the decoding process.

[0248] In the embodiments of this specification, as a specific example, the first decoding resource is a hardware decoding resource, and the second decoding resource is a software decoding resource. As another specific example, the first decoding resource is a graphics processing unit (GPU) decoding chip, and the second decoding resource is a central processing unit (CPU) decoding chip.

[0249] After decoding the video compression frames in the above-mentioned embodiment, the image reconstruction of the virtual viewpoint can be performed based on the free viewpoint according to the decoded image data. Based on the video compression frames of the above-mentioned different encoding methods, the specific data referenced by the image reconstruction are slightly different, and the following two specific embodiments are used for example:

[0250] Reference Fig.24 The flowchart of the video processing method shown in the figure shows that for a video frame obtained by using a custom depth map decoding method, the image can be reconstructed in the following manner:

[0251] S241, obtaining viewpoint parameter data from the frame header of the video compression frame.

[0252] S242, obtaining a virtual viewpoint, and based on the parameter data of the viewpoint, selecting a texture map in a corresponding texture map sub-region and a depth map in a depth map sub-region in the texture map stitched image and the depth map stitched image that match the virtual viewpoint position as a reference texture map and a reference depth map.

[0253] In a specific implementation, the virtual viewpoint may be determined based on user interaction behavior, or based on pre-set virtual viewpoint position information.

[0254] In some embodiments of the present specification, in order to reduce the amount of data calculation while ensuring the quality of the reconstructed image, only the texture map and depth map of some viewpoints can be selected to reconstruct the image of the virtual viewpoint based on the spatial position relationship information. Specifically, the reference texture map and the reference depth map can be determined based on the distance relationship between the virtual viewpoint position and the texture map and the depth map in each texture map sub-area in the texture map spliced ​​image and the depth map spliced ​​image.

[0255] The spatial position relationship information can be determined based on parameter information of the texture map and the depth map, and the parameter information can be stored in a frame header of a video compression frame or a channel header of a video channel. The parameter information can include intrinsic parameter information and extrinsic parameter information of each viewpoint.

[0256] S243: reconstructing an image of the virtual viewpoint based on the reference texture map and the reference depth map.

[0257] In a specific implementation, the image of the virtual viewpoint can be obtained by performing combined rendering based on the reference texture map and the reference depth map.

[0258] The above method is used to reconstruct the image of the virtual viewpoint. Since the texture map stitching image and the depth map stitching image can be encoded separately using a coding method that matches the image characteristics, the compression loss of the decoded texture map stitching image and the depth map stitching image is small, so the quality of the reconstructed virtual viewpoint image can be improved.

[0259] Reference Fig.25 The flowchart of the video processing method shown in the figure shows that for a video frame obtained by multi-view joint decoding, the image can be reconstructed in the following manner:

[0260] S251, obtaining a viewpoint identifier, a frame identifier of the video compression frame, and parameter data of each viewpoint from a frame header of the video compression frame.

[0261] S252, obtaining a virtual viewpoint, and based on the identification and parameter data of each viewpoint, selecting a texture map and a depth map matching the virtual viewpoint from the texture maps and corresponding depth maps of the synchronized multiple viewpoints as a reference texture map and a reference depth map.

[0262] In a specific implementation, the virtual viewpoint may be determined based on user interaction behavior, or based on pre-set virtual viewpoint position information.

[0263] In some embodiments of the present specification, in order to reduce the amount of data calculation while ensuring the quality of the reconstructed image, only the texture map and depth map of some viewpoints can be selected to reconstruct the image of the virtual viewpoint based on the spatial position relationship information. Specifically, the reference texture map and the reference depth map can be determined based on the distance relationship between the virtual viewpoint position and the texture map and the depth map in each texture map sub-area in the texture map spliced ​​image and the depth map spliced ​​image.

[0264] The spatial position relationship information can be determined based on parameter information of the texture map and the depth map, and the parameter information can be stored in a frame header of a video compression frame or a channel header of a video channel. The parameter information can include intrinsic parameter information and extrinsic parameter information of each viewpoint.

[0265] S253: reconstructing the image of the virtual viewpoint based on the reference texture map and the reference depth map.

[0266] In a specific implementation, the image of the virtual viewpoint can be obtained by performing combined rendering based on the reference texture map and the reference depth map.

[0267] The above method is used to reconstruct the image of the virtual viewpoint. Since the texture map and depth map of each viewpoint can be encoded separately in a coding method that matches the image characteristics, the compression loss of the texture map and depth map of each viewpoint obtained by decoding is small, so the quality of the reconstructed virtual viewpoint image can be improved.

[0268] In a specific implementation, the free viewpoint image obtained by reconstructing the virtual viewpoint may be further processed. An exemplary expansion method is given below.

[0269] In order to enrich the user's visual experience, augmented reality (AR) special effects can be embedded in the reconstructed free viewpoint image. Fig.26 The flowchart of the video processing method shown in the figure implements the implantation of AR special effects in the following manner:

[0270] S261, obtaining a virtual rendering target object in the image of the virtual viewpoint.

[0271] In a specific implementation, certain objects in the image of the free viewpoint video can be determined as virtual rendering target objects based on certain indication information, and the indication information can be generated based on user interaction, or can be obtained based on certain preset trigger conditions or third-party instructions. In an optional embodiment of the present specification, in response to a special effect generation interactive control instruction, a virtual rendering target object in the image of the virtual viewpoint can be obtained.

[0272] S262, obtaining a virtual information image generated based on the augmented reality special effect input data of the virtual rendering target object.

[0273] In the embodiment of the present specification, the implanted AR special effects are presented in the form of a virtual information image. The virtual information image can be generated based on the augmented reality special effects input data of the target object. After determining the virtual rendering target object, the virtual information image generated based on the augmented reality special effects input data of the virtual rendering target object can be obtained.

[0274] In the embodiments of the present specification, the virtual information image corresponding to the virtual rendering target object may be generated in advance, or may be generated immediately in response to a special effect generation instruction.

[0275] In a specific implementation, a virtual information image that matches the position of the virtual rendering target object can be obtained based on the position of the virtual rendering target object in the reconstructed image obtained by three-dimensional calibration, so that the obtained virtual information image can be more closely matched with the position of the virtual rendering target object in the three-dimensional space, and the displayed virtual information image is more consistent with the real state in the three-dimensional space, so that the displayed composite image is more realistic and vivid, thereby enhancing the user's visual experience.

[0276] In a specific implementation, based on the augmented reality special effect input data of the virtual rendering target object, a virtual information image corresponding to the target object can be generated according to a preset special effect generation method.

[0277] In specific implementations, a variety of special effect generation methods can be used.

[0278] For example, the augmented reality special effect input data of the target object may be input into a preset three-dimensional model, and based on the position of the virtual rendering target object in the image obtained by three-dimensional calibration, a virtual information image matching the virtual rendering target object may be output;

[0279] For example, the augmented reality special effects input data of the virtual rendering target object can be input into a preset machine learning model, and based on the position of the virtual rendering target object in the image obtained by three-dimensional calibration, a virtual information image matching the virtual rendering target object is output.

[0280] S263, synthesizing and displaying the virtual information image and the image of the virtual viewpoint.

[0281] In a specific implementation, there are multiple ways to synthesize and display the virtual information image and the image of the virtual viewpoint. Two specific achievable examples are given below:

[0282] Example 1: Fusing the virtual information image with the corresponding image to obtain a fused image, and displaying the fused image;

[0283] Example 2: superimposing the virtual information image on the corresponding image to obtain a superimposed composite image, and displaying the superimposed composite image.

[0284] In a specific implementation, the obtained composite image can be directly displayed, or the obtained composite image can be inserted into the video stream to be played for display. For example, the fused image can be inserted into the video stream to be played for display.

[0285] The free viewpoint video may include a special effects display identifier. In a specific implementation, the superimposition position of the virtual information image in the image of the virtual viewpoint may be determined based on the special effects display identifier, and then the virtual information image may be superimposed and displayed at the determined superimposition position.

[0286] In order to enable those skilled in the art to better understand and implement the embodiments of this specification, a video processing device that can implement the above-mentioned video processing method is described below.

[0287] Reference Fig. 27The structural diagram of the video processing device shown in FIG. Fig.26 As shown, the video processing device 270 may include: a texture map stitching unit 271, a depth map stitching unit 272, a first encoding unit 273 and a second encoding unit 274, wherein:

[0288] The texture map stitching unit 271 is adapted to stitch the texture maps of multiple viewpoints of frame synchronization into a texture map stitching image;

[0289] The depth map stitching unit 272 is adapted to stitch the depth maps corresponding to the texture maps of the multiple viewpoints of the frame synchronization into a depth map stitching image;

[0290] The first encoding unit 273 is adapted to encode the texture map splicing image to obtain a corresponding texture map compressed image;

[0291] The second encoding unit 274 is adapted to encode the depth map stitching image to obtain a corresponding depth map compressed image.

[0292] By using the video processing device, the texture maps and depth maps of multiple viewpoints synchronized to form a free viewpoint video can be compressed and encoded in a manner of custom encoding of the depth map.

[0293] Reference Fig.28 The structural diagram of the video processing device shown in FIG. 1 is a schematic diagram of the structure of the video processing device shown in FIG. 1 . The embodiment of this specification also provides another video processing device, such as Fig.28 As shown, the video processing device 280 may include: an encoding category determination unit 281 and a first encoding processing unit 282, wherein:

[0294] A coding category determination unit 281, adapted to determine the coding category to which the coding mode belongs;

[0295] The first encoding processing unit 282 is suitable for storing the encoding category identifier and encoding identifier information corresponding to the encoding method in the encoding information area in the image set user information when it is determined that the encoding method belongs to the depth map custom encoding; and splicing the texture maps of multiple viewpoints synchronized by frames into a texture map splicing image, splicing the depth maps corresponding to the texture maps of multiple viewpoints synchronized by frames into a depth map splicing image, and encoding the texture map splicing image and the depth map splicing image respectively to obtain the corresponding texture map compressed image and depth map compressed image.

[0296] In a specific implementation, the video processing device 280 may also include: a second encoding processing unit 283, which is suitable for obtaining the viewpoint identifier and frame identifier of the texture map and depth map of the multiple viewpoints synchronized with the frame when it is determined that the encoding method belongs to multi-view joint encoding, and storing them in the image information area in the image set user information, and storing the encoding identification information corresponding to the encoding method in the encoding information area in the image set user information; and encoding the texture map and depth map of the multiple viewpoints synchronized with the frame respectively using a preset multi-view joint encoding method to obtain a texture map compressed image and a depth map compressed image.

[0297] In the specific implementation, Fig.28 As shown, the video processing device 280 may further include: an encapsulation unit 284, adapted to encapsulate the texture map compressed image and the depth map compressed image in the same video channel.

[0298] As an optional example, the encapsulation unit 284 may place the texture map compressed image and the depth map compressed image obtained by custom encoding of the depth map in the frame body of the video compression frame, and place the depth map compressed image in the frame header of the video compression frame.

[0299] As an optional example, the encapsulation unit 284 may interleave the texture map compressed image and the depth map compressed image obtained by multi-view joint coding and place them in the frame body of the video compression frame.

[0300] The embodiment of this specification also provides another video processing device, which can decapsulate and decode video compression frames. Fig.29 As shown, the video processing device 290 may include: a decapsulation unit 291 and a first decoding unit 292, wherein:

[0301] The decapsulation unit 291 is adapted to decapsulate the free viewpoint video stream to obtain the video compression frame and the encoding category information to which the encoding method of the video compression frame belongs;

[0302] The first decoding unit 292 is suitable for decoding the texture map compressed image and the depth map compressed image in the video compression frame respectively based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts custom depth map encoding, to obtain a texture map stitching image and a depth map stitching image; wherein the texture map stitching image includes texture maps of multiple synchronized viewpoints; and the depth map stitching image includes depth maps corresponding to the texture maps of the multiple viewpoints.

[0303] In the specific implementation, continue to refer to Fig.29The video processing device may also include: a second decoding unit 293, which is suitable for decoding the texture map compressed image and the depth map compressed image in the video compression frame based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts multi-view joint encoding, using the corresponding multi-view joint decoding method to obtain the texture map of multiple viewpoints synchronized and the corresponding depth map.

[0304] Reference Fig.30 Another video processing device shown in the embodiment of this specification is as follows: Fig.30 The video processing device 300 shown may include: an indication information decoding unit 301, a texture map decoding unit 302 and a depth map decoding unit 303, wherein:

[0305] The indication information decoding unit 301 is adapted to decode the frame header of the video compression frame, and obtain the encoding identification information of the depth map compression image and the encoding identification information of the texture map compression image in the video compression frame from the frame header;

[0306] The texture map decoding unit 302 is adapted to decode the texture map compressed image in a decoding manner corresponding to the coding identification information of the texture map compressed image to obtain a texture map spliced ​​image, wherein the depth map spliced ​​image includes texture maps of multiple synchronized viewpoints;

[0307] The depth map decoding unit 303 is adapted to decode the depth map compressed image using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain a depth map stitching image, wherein the depth map stitching image includes depth maps corresponding to the texture map viewpoints of the multiple viewpoints.

[0308] The embodiment of this specification also provides a corresponding video processing device, referring to Fig.31 The electronic device shown in FIG. 1 is a schematic diagram of a structure of an electronic device. In the present embodiment, the electronic device can be used to encode and compress video data, such as Fig.31 As shown, the electronic device 310 may include:

[0309] The image processing device 311 is adapted to stitch the texture maps of multiple viewpoints of frame synchronization into a texture map stitching image, and stitch the depth maps corresponding to the texture maps of multiple viewpoints of frame synchronization into a depth map stitching image;

[0310] The first encoding device 312 is adapted to encode the texture map splicing image to obtain a corresponding texture map compressed image;

[0311] The second encoding device 313 is adapted to encode the depth map spliced ​​image to obtain a corresponding depth map compressed image.

[0312] In a specific implementation, the image processing device 311 can be implemented by a preset data processing chip, for example, it can be any one of a single-core or multi-core processor, a GPU chip, an FPGA chip, etc.

[0313] In a specific implementation, the first encoding device 312 and the second encoding device 313 can be implemented by a special encoder, encoding software, or encoding software and an encoder in collaboration.

[0314] In a specific implementation, the first encoding device 312 and the second encoding device 313 can be executed independently or in coordination.

[0315] In a specific implementation, the first encoding device 312 and the second encoding device 313 may also be used to perform multi-view joint encoding, for example, the first encoding device encodes the texture map, and the second encoding device encodes the depth map. The specific multi-view joint encoding processing method can refer to the above embodiment, which will not be described here.

[0316] Reference Fig.32 The structural diagram of another electronic device shown in FIG. 1 can be used to decode the video compression frame of the video channel, such as Fig.32 As shown, the electronic device 320 includes: a first decoding device 321 and a second decoding device 322, wherein:

[0317] The first decoding device 321 is adapted to decode a frame header of a video compression frame, and when the encoding identification information obtained from the image set user information of the frame header includes the encoding identification information of a depth map compression image and the encoding identification information of a texture map compression image, the texture map compression image is decoded using a decoding method corresponding to the encoding identification information of the texture map compression image to obtain a texture map splicing image, wherein the texture map splicing image includes texture maps of multiple synchronized viewpoints, and the second decoder is triggered to decode the depth map compression image;

[0318] The second decoding device 322 is suitable for decoding the depth map compressed image using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain a depth map stitching image, wherein the depth map stitching image includes depth maps corresponding to the texture map viewpoints of the multiple viewpoints.

[0319] By using the electronic device, it is possible to decode the video compression frames obtained by using the depth map custom decoding method.

[0320] In a specific implementation, the above electronic device may also be used to decode the video compression frame obtained by the multi-view joint encoding method, wherein the first decoding device 321 may perform texture map encoding, and the second decoding device 322 may perform depth map encoding. The specific multi-view joint decoding execution process may refer to the introduction of the above embodiment, which will not be repeated here.

[0321] Reference Fig.33 The structural diagram of another electronic device shown in FIG. 1 is a schematic diagram of a structure of another electronic device, and the embodiment of this specification also provides another electronic device, wherein, as Fig.33 As shown, the electronic device 330 may include a memory 331 and a processor 332, wherein the memory 331 stores computer instructions that can be executed on the processor 332, wherein the processor can execute the steps of the method described in any of the aforementioned embodiments when executing the computer instructions.

[0322] Based on the location of the electronic device in the entire video processing system, the electronic device may further include other electronic components or assemblies.

[0323] For example, continue to refer to Fig.33 The electronic device 330 may also include a communication component 333, which can communicate with the acquisition system or the cloud server to obtain a texture map of multiple viewpoints synchronized for generating free viewpoint video frames, or obtain a free viewpoint video compression frame obtained by encoding and packaging using the video processing method of the aforementioned embodiment of this specification. The processor 332 can then perform decapsulation and decoding processing based on the video compression frame obtained by the communication component 333, and reconstruct the free viewpoint video of the virtual viewpoint according to the virtual viewpoint position.

[0324] For example, in some electronic devices, continue to refer to Fig.33 The electronic device 330 may further include a display component 334 (such as a display, a touch screen, a projector) to display the reconstructed image of the virtual viewpoint.

[0325] As a server device, the electronic device can be set as a cloud server or server cluster, or as a local server to compress the free viewpoint video frame before transmission. As a terminal device, the electronic device 330 can be a handheld electronic device such as a mobile phone, a laptop computer, a desktop computer, a set-top box, and other electronic devices with video processing and playback functions. The terminal device can be used to decode the received video compression frame, and reconstruct the image of the virtual viewpoint based on the decoded video frame and the acquired virtual viewpoint. The virtual viewpoint can be determined based on user interaction behavior or based on pre-set virtual viewpoint position information.

[0326] In some embodiments of the present specification, the memory, processor, communication component, and display component may communicate with each other via a bus network.

[0327] In the specific implementation, Fig.33 As shown, the communication component 333 and the display component 334 etc. may be components disposed inside the electronic device 330, or may be external devices connected via expansion components such as an expansion interface, an expansion dock, and an expansion cable.

[0328] In a specific implementation, the processor 332 can be implemented collaboratively through any one or more of a central processing unit (CPU) (such as a single-core processor, a multi-core processor), a CPU group, a graphics processing unit (GPU), an AI chip, an FPGA chip, etc.

[0329] In a specific implementation, for a large number of free viewpoint video frames, in order to reduce processing delay, an electronic device cluster consisting of multiple electronic devices can also be used for collaborative implementation.

[0330] In order to enable those skilled in the art to better understand and implement, a specific application scenario is used below for illustration. Fig.34 The structural diagram of the video processing system shown in FIG. Fig.34 As shown, it is a structural diagram of a video processing system in an application scenario, wherein a layout scenario of a data processing system for a basketball game is shown. The video processing system A0 includes an acquisition array A1 composed of multiple acquisition devices, a data processing device A2, a cloud server cluster A3, a playback control device A4, a playback terminal A5 and an interactive terminal A6.

[0331] Reference Fig.34 , with the basketball hoop on the left as the core viewing point, and the fan-shaped area with the core viewing point as the center of the circle and located on the same plane as the core viewing point as the preset multi-angle free viewing angle range. The acquisition devices in the acquisition array A1 can be placed in different positions of the on-site acquisition area in a fan-shaped manner according to the preset multi-angle free viewing angle range, and can respectively and simultaneously acquire video data streams from corresponding angles in real time.

[0332] In a specific implementation, the acquisition equipment can also be set in the ceiling area of ​​a basketball stadium, on a basketball stand, etc. The acquisition equipment can be arranged and distributed along a straight line, a fan, an arc, a circle or an irregular shape. The specific arrangement can be set according to one or more factors such as the specific on-site environment, the number of acquisition equipment, the characteristics of the acquisition equipment, the imaging effect requirements, etc. The acquisition equipment can be any device with a camera function, such as an ordinary camera, a mobile phone, a professional camera, etc.

[0333] In order not to affect the work of the acquisition equipment, the data processing device A2 can be placed in a non-acquisition area on site and can be regarded as an on-site server. The data processing device A2 can send a stream pulling instruction to each acquisition device in the acquisition array A1 through a wireless local area network. Based on the stream pulling instruction sent by the data processing device A2, each acquisition device in the acquisition array A1 transmits the obtained video data stream to the data processing device A2 in real time. Among them, each acquisition device in the acquisition array A1 can transmit the obtained video data stream to the data processing device A2 in real time through the switch A7. The acquisition array A1 and the switch A7 together form an acquisition system.

[0334] When the data processing device A2 receives the video frame capture instruction, it captures the video frame at the specified frame time from the received multi-channel video data streams to obtain frame images of multiple synchronous video frames, and uploads the obtained multiple synchronous video frames at the specified frame time to the server cluster A3 in the cloud.

[0335] Correspondingly, the server cluster A3 in the cloud takes the original texture maps of the received multiple synchronized video frames as an image combination, determines the parameter data corresponding to the image combination and the estimated depth map corresponding to each original texture map in the image combination, and based on the parameter data corresponding to the image combination, the pixel data of the texture map in the image combination and the depth data of the corresponding depth map, reconstructs the frame image based on the acquired virtual viewpoint path to obtain corresponding multi-angle free-viewpoint video data.

[0336] The server may be placed in the cloud, and in order to process data in parallel more quickly, a server cluster A3 in the cloud may be formed by a plurality of different servers or server groups according to different data to be processed.

[0337] For example, the cloud server cluster A3 may include: a first cloud server A31, a second cloud server A32, a third cloud server A33, and a fourth cloud server A34. The first cloud server A31 may be used to determine the parameter data corresponding to the image combination; the second cloud server A32 may be used to determine the estimated depth map of the original texture map of each viewpoint in the image combination and perform depth map correction processing; the third cloud server A33 may perform frame image reconstruction based on the position information of the virtual viewpoint, the parameter data corresponding to the image combination, the texture map and the depth map of the image combination, and the depth image based on the depth image, using a virtual viewpoint reconstruction (Depth Image Based Rendering, DIBR) algorithm to obtain an image of the virtual viewpoint; the fourth cloud server A34 may be used to generate free viewpoint video (multi-angle free viewpoint video).

[0338] It is understandable that the first cloud server A31, the second cloud server A32, the third cloud server A33, and the fourth cloud server A34 may also be a server group composed of a server array or a server sub-cluster, and the embodiment of the present invention does not limit this.

[0339] Then, the playback control device A4 can insert the received free viewpoint video frame into the video stream to be played, and the playback terminal A5 receives the video stream to be played from the playback control device A4 and plays it in real time. The playback control device A4 can be a manual playback control device or a virtual playback control device. In a specific implementation, a dedicated server that can automatically switch video streams can be set as a virtual playback control device to control the data source. A director control device such as a director station can be used as a playback control device in an embodiment of the present invention.

[0340] It can be understood that the data processing device A2 can be placed in a non-collection area on site or in the cloud according to the specific scenario, and the server (cluster) and playback control device can be placed in a non-collection area on site, in the cloud or on the terminal access side according to the specific scenario. This embodiment is not used to limit the specific implementation and protection scope of the present invention.

[0341] The data processing device A2 or the server cluster A3 in the cloud can use the video processing method in the embodiment of this specification to encode and encapsulate the texture map of each viewpoint and the corresponding depth map; the playback terminal A5 and the interactive terminal A6 can decapsulate and decode the received video compression frame. The playback terminal A5 and the interactive terminal A6 can be specially equipped with corresponding decoding chips, decoding modules or decoding software, etc., at least one or more combinations of which can perform decoding processing of video compression frames. For specific implementation, please refer to the video processing method example of the decoding end mentioned above.

[0342] The embodiments of the present specification also provide a computer-readable storage medium on which computer instructions are stored, wherein the computer instructions, when executed, execute the steps of the method described in any of the aforementioned embodiments. For details, please refer to the introduction of the aforementioned embodiments, which will not be repeated here.

[0343] In a specific implementation, the computer-readable storage medium may be any appropriate readable storage medium such as an optical disk, a mechanical hard disk, a solid-state hard disk, etc.

[0344] The explanation of terms, principles, specific implementations and beneficial effects involved in the multi-video processing device, electronic device, storage medium, etc. in the embodiments of the present invention can be found in the video processing method in the embodiments of the present invention, and will not be repeated here.

[0345] Although the embodiments of this specification are disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this specification, so the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A video processing method, wherein: include: splicing texture maps of multiple viewpoints synchronized by frames into a texture map splicing image; splicing the depth maps corresponding to the texture maps of the multiple viewpoints of the frame synchronization into a depth map splicing image; The texture map stitching image and the depth map stitching image are respectively encoded to obtain corresponding texture map compressed image and depth map compressed image; wherein, the texture map stitching image is encoded using a first encoding method to obtain a texture map compressed image, and the depth map stitching image is encoded using a second encoding method to obtain a depth map compressed image, wherein the second encoding method is an encoding method based on a region of interest, and the second encoding method is different from the first encoding method.

2. The method according to claim 1, wherein: Also includes: The texture map compressed image and the depth map compressed image are encapsulated in a video compression frame of the same video channel.

3. The method according to claim 2, wherein: The depth map compressed image is encapsulated in a frame header area of ​​the video compression frame.

4. The method according to claim 1, wherein: The depth map stitching image is encoded using a region of interest based encoding method to obtain a corresponding depth map compressed image.

5. The method according to claim 4, wherein: The ROI pixel area in the depth map stitched image is encoded based on a preset constant quantization parameter to obtain a corresponding depth map compressed image.

6. The method according to claim 5, wherein: A first constant quantization parameter is used to encode the foreground edge pixel area in the depth map sub-area corresponding to each viewpoint contained in the depth map stitched image, and a second constant quantization parameter is used to encode the non-foreground edge pixel area in the depth map sub-area corresponding to each viewpoint, and the parameter value of the first constant quantization parameter is less than the parameter value of the second constant quantization parameter.

7. The method according to any one of claims 1 to 6, wherein: The pixel points of the depth map sub-region corresponding to each viewpoint contained in the depth map stitched image are all or part of the pixel points in a set of pixel points in the texture map stitched image that correspond one-to-one to the pixel points in the texture map sub-region of the corresponding viewpoint.

8. A video processing method, wherein: include: Determine the coding category to which the coding method belongs; If it is determined that the encoding method belongs to custom encoding of the depth map, the encoding identification information corresponding to the encoding method is stored in the encoding information area in the image set user information; and the texture maps of multiple viewpoints of frame synchronization are spliced ​​into a texture map spliced ​​image, and the depth maps corresponding to the texture maps of multiple viewpoints of frame synchronization are spliced ​​into a depth map spliced ​​image, and the texture map spliced ​​image and the depth map spliced ​​image are respectively encoded to obtain the corresponding texture map compressed image and depth map compressed image; wherein, the texture map spliced ​​image is encoded using a first encoding method to obtain a texture map compressed image, and the depth map spliced ​​image is encoded using a second encoding method to obtain a depth map compressed image, and the second encoding method is encoded based on an encoding method of a region of interest, and the second encoding method is different from the first encoding method.

9. The method according to claim 8, wherein: Also includes: The texture map compressed image and the depth map compressed image are encapsulated in the same video channel.

10. The method according to claim 9, wherein: The step of encapsulating the texture map compressed image and the depth map compressed image in the same video channel comprises: The depth map compressed image is encapsulated into a depth map region in a frame header of a video compression frame of the same video channel.

11. The method according to any one of claims 8 to 10, wherein: Also includes: If it is determined that the encoding method belongs to multi-view joint encoding, obtaining viewpoint identifiers and frame identifiers of the texture maps and depth maps of the multiple viewpoints of the frame synchronization, and storing them in the image information area in the image set user information, and storing the encoding identifier information corresponding to the encoding method in the encoding information area in the image set user information; And the texture maps and depth maps of the frame-synchronized multiple viewpoints are respectively encoded using a preset multi-view joint encoding method to obtain a texture map compressed image and a depth map compressed image.

12. The method according to claim 11, wherein: Also includes: The texture map compressed image and the depth map compressed image are combined as an image and encapsulated into the same video channel to obtain a video compression frame including the texture map compressed image and the depth map compressed image.

13. The method according to claim 11, wherein: The acquiring of viewpoint identifiers and frame identifiers of the texture maps and depth maps of the multiple viewpoints of the frame synchronization and storing them in the image information area in the image set user information, and storing the encoding identifier information corresponding to the encoding method in the encoding information area in the image set user information, comprises: The coding category identifier corresponding to the coding mode is stored in the coding category area in the channel header of the video channel; The coding identification information is stored in the coding information area in the video compression frame header; The viewpoint identifiers and frame identifiers of the texture maps and depth maps of the multiple viewpoints of the frame synchronization are stored in the image information area in the frame header of the video compression frame according to the acquisition order.

14. The method according to any one of claims 8 to 10, wherein: The step of stitching the texture maps of multiple viewpoints synchronized by frames into a texture map stitching image, and stitching the depth maps corresponding to the texture maps of multiple viewpoints synchronized by frames into a depth map stitching image comprises: Scanning the texture maps of the multiple viewpoints of the frame synchronization in a raster scanning manner, and splicing the texture maps of the multiple viewpoints in a raster scanning order to obtain the texture map splicing image; The depth maps corresponding to the texture maps of the multiple viewpoints of the frame synchronization are scanned in the raster scanning manner, and the depth maps of the multiple viewpoints are spliced ​​in a raster scanning order to obtain the depth map spliced ​​image.

15. A video processing method, wherein: include: Decapsulating the free viewpoint video stream to obtain video compression frames and encoding category information of the encoding method of the video compression frames; Based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts custom depth map encoding, the texture map compressed image and the depth map compressed image in the video compression frame are decoded respectively to obtain a texture map stitching image and a depth map stitching image; wherein the texture map stitching image includes texture maps of multiple synchronized viewpoints; the depth map stitching image includes depth maps corresponding to the texture maps of the multiple viewpoints, wherein the depth map compressed image is decoded using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain the depth map stitching image, and the texture map compressed image is decoded using a decoding method corresponding to the encoding identification information of the texture map compressed image to obtain the texture map stitching image, and the decoding method corresponding to the encoding identification information of the texture map compressed image and the decoding method corresponding to the encoding identification information of the depth map compressed image are different.

16. The method according to claim 15, wherein: The decoding of the texture map compressed image and the depth map compressed image in the video compression frame to obtain the texture map stitching image and the depth map stitching image includes: Decoding a frame header of the video compression frame, and acquiring coding identification information of the texture map compression image and coding identification information of the depth map compression image from a coding information storage area in the frame header; Decoding the depth map compressed image using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain the depth map stitched image; The texture map compressed image is decoded using a decoding method corresponding to the coding identification information of the texture map compressed image to obtain the texture map spliced ​​image.

17. The method according to claim 16, wherein: The decoding of the texture map compressed image and the depth map compressed image in the video compression frame to obtain the texture map stitching image and the depth map stitching image includes: Decoding the texture map compressed image using a first decoding resource to obtain a texture map spliced ​​image; The depth map compressed image is decoded using a second decoding resource to obtain a depth map stitched image.

18. The method according to claim 16, wherein: Also includes: Acquire parameter data of a viewpoint from a frame header of the video compression frame; Acquire a virtual viewpoint, and based on the parameter data of the viewpoint, select a texture map in a corresponding texture map sub-region and a depth map in a depth map sub-region in the texture map stitched image and the depth map stitched image that match the position of the virtual viewpoint as a reference texture map and a reference depth map; Based on the reference texture map and the reference depth map, an image of the virtual viewpoint is reconstructed.

19. The method according to any one of claims 15 to 18, wherein: Also includes: Based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts multi-view joint encoding, the texture map compressed image and the depth map compressed image in the video compression frame are decoded using the corresponding multi-view joint decoding method to obtain synchronized texture maps of multiple viewpoints and corresponding depth maps.

20. The method according to claim 19, wherein: The decoding of the joint compressed image in the video compression frame to obtain texture maps of multiple synchronized viewpoints and corresponding depth maps includes: Decoding the frame header of the video compression frame, and acquiring the coding identification information of the texture map compression image and the coding identification of the depth map compression image from the coding information storage area in the frame header; The texture map compressed image and the depth map compressed image are decoded using a multi-view joint decoding method corresponding to the coding identification information of the texture map compressed image and the coding identification of the depth map compressed image to obtain the texture maps of the synchronized multiple viewpoints and the corresponding depth maps.

21. The method according to claim 20, wherein: Also includes: Obtaining the viewpoint identifier, the frame identifier of the video compression frame, and parameter data of each viewpoint from the frame header of the video compression frame; Acquire a virtual viewpoint, and select, based on the identifier and parameter data of each viewpoint, a texture map and a depth map matching the virtual viewpoint from the texture maps and corresponding depth maps of the synchronized multiple viewpoints as a reference texture map and a reference depth map; Based on the reference texture map and the reference depth map, an image of the virtual viewpoint is reconstructed.

22. A video processing method, wherein: include: Decoding a frame header of a video compression frame, and obtaining, from the frame header, coding identification information of a depth map compression image and coding identification information of a texture map compression image included in the video compression frame; Decoding the texture map compressed image using a decoding method corresponding to the coding identification information of the texture map compressed image to obtain a texture map spliced ​​image, wherein the texture map spliced ​​image includes texture maps of multiple synchronized viewpoints; The depth map compressed image is decoded using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain a depth map stitching image, wherein the depth map stitching image includes depth maps corresponding to texture map viewpoints of the multiple viewpoints, and the decoding method corresponding to the encoding identification information of the texture map compressed image is different from the decoding method corresponding to the encoding identification information of the depth map compressed image.

23. A video processing device, wherein: include: A texture map stitching unit, adapted to stitch the texture maps of multiple viewpoints of frame synchronization into a texture map stitching image; A depth map stitching unit, adapted to stitch the depth maps corresponding to the texture maps of the plurality of viewpoints of the frame synchronization into a depth map stitching image; A first encoding unit, adapted to encode the texture map splicing image using a first encoding method to obtain a corresponding texture map compressed image; The second encoding unit is adapted to encode the depth map stitching image using a second encoding method to obtain a corresponding depth map compressed image, wherein the second encoding method is an encoding method based on a region of interest, and the second encoding method is different from the first encoding method.

24. A video processing device, wherein: include: A coding category determination unit, adapted to determine the coding category to which the coding method belongs; The first encoding processing unit is suitable for storing the encoding identification information corresponding to the encoding method into the encoding information area in the image set user information when it is determined that the encoding method belongs to the depth map custom encoding; and stitching the texture maps of multiple viewpoints of frame synchronization into a texture map stitching image, stitching the depth maps corresponding to the texture maps of multiple viewpoints of frame synchronization into a depth map stitching image, and encoding the texture map stitching image and the depth map stitching image respectively to obtain the corresponding texture map compressed image and depth map compressed image; wherein, the texture map stitching image is encoded using a first encoding method to obtain a texture map compressed image, and the depth map stitching image is encoded using a second encoding method to obtain a depth map compressed image, and the second encoding method is encoded based on an encoding method of a region of interest, and the second encoding method is different from the first encoding method.

25. The device according to claim 24, wherein: Also includes: The encapsulation unit is adapted to encapsulate the texture map compressed image and the depth map compressed image in the same video channel.

26. A video processing device, wherein: include: A decapsulation unit, adapted to decapsulate the free viewpoint video stream to obtain a video compression frame and encoding category information to which the encoding method of the video compression frame belongs; The first decoding unit is adapted to decode the texture map compressed image and the depth map compressed image in the video compression frame respectively based on the encoding category information of the video compression frame, when it is determined that the video compression frame adopts custom depth map encoding, to obtain a texture map stitching image and a depth map stitching image; wherein the texture map stitching image includes texture maps of multiple synchronized viewpoints; the depth map stitching image includes depth maps corresponding to the texture maps of the multiple viewpoints, the depth map compressed image is decoded using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain the depth map stitching image, the texture map compressed image is decoded using a decoding method corresponding to the encoding identification information of the texture map compressed image to obtain the texture map stitching image, and the decoding method corresponding to the encoding identification information of the texture map compressed image and the decoding method corresponding to the encoding identification information of the depth map compressed image are different.

27. The device according to claim 26, wherein: Also includes: The second decoding unit is suitable for decoding the joint compressed image in the video compression frame using the corresponding multi-view joint decoding method based on the encoding category information of the video compression frame to obtain texture maps of multiple synchronized viewpoints and corresponding depth maps.

28. A video processing device, wherein: include: an indication information decoding unit, adapted to decode a frame header of a video compression frame, and obtain, from the frame header, encoding identification information of a depth map compression image and encoding identification information of a texture map compression image included in the video compression frame; A texture map decoding unit, adapted to decode the texture map compressed image in a decoding manner corresponding to the coding identification information of the texture map compressed image to obtain a texture map spliced ​​image, wherein the texture map spliced ​​image includes texture maps of multiple synchronized viewpoints; A depth map decoding unit is suitable for decoding the depth map compressed image using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain a depth map stitching image, wherein the depth map stitching image includes a depth map corresponding to the texture map viewpoint of the multiple viewpoints, and the decoding method corresponding to the encoding identification information of the texture map compressed image and the decoding method corresponding to the encoding identification information of the depth map compressed image are different.

29. An electronic device, wherein: include: An image processing device adapted to stitch texture maps of multiple viewpoints of frame synchronization into a texture map stitched image, and stitch depth maps corresponding to the texture maps of multiple viewpoints of frame synchronization into a depth map stitched image; A first encoding device, adapted to encode the texture map splicing image using a first encoding method to obtain a corresponding texture map compressed image; The second encoding device is suitable for encoding the depth map stitching image using a second encoding method to obtain a corresponding depth map compressed image. The second encoding method is an encoding method based on a region of interest, and the second encoding method is different from the first encoding method.

30. An electronic device, wherein: include: A first decoding device and a second decoding device, wherein: The first decoding device is adapted to decode a frame header of a video compression frame, and when the encoding identification information obtained from the image set user information of the frame header includes the encoding identification information of a depth map compression image and the encoding identification information of a texture map compression image, the texture map compression image is decoded using a decoding method corresponding to the encoding identification information of the texture map compression image to obtain a texture map splicing image, wherein the texture map splicing image includes texture maps of multiple synchronized viewpoints, and the second decoding device is triggered to decode the depth map compression image; The second decoding device is suitable for decoding the depth map compressed image using a decoding method corresponding to the encoding identification information of the depth map compressed image to obtain a depth map stitching image, wherein the depth map stitching image includes a depth map corresponding to the texture map viewpoint of the multiple viewpoints, and the decoding method corresponding to the encoding identification information of the texture map compressed image and the decoding method corresponding to the encoding identification information of the depth map compressed image are different.

31. An electronic device comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, wherein: When the processor executes the computer instructions, the processor performs the steps of the method described in any one of claims 1 to 7, any one of claims 8 to 14, any one of claims 15 to 21, or claim 22.

32. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed, the steps of the method described in any one of claims 1 to 7, any one of claims 8 to 14, any one of claims 15 to 21, or claim 22 are executed.

Images