Video encoding methods, video decoding methods and related devices
By dividing panoramic video images into image blocks and generating reference information for independent and non-independent coded viewpoints, the problems of computational resources and bandwidth consumption during panoramic video encoding and decoding are solved, enabling more efficient video transmission and playback.
Patent Information
- Application Number
- CN202210557264.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-20
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-05-20
AI Technical Summary
During panoramic video encoding and decoding, videos from multiple perspectives need to be encoded and compressed independently, resulting in high consumption of device computing resources and increased power consumption. Furthermore, excessive bandwidth is consumed during transmission, affecting the smoothness of video playback.
Video images from multiple perspectives at the same time are divided into N image blocks, generating encoded data that includes encoded information from independent encoded perspectives and reference information from non-independent encoded perspectives. The reference information from non-independent encoded perspectives is used to instruct them to decode by referring to the encoded information from independent encoded perspectives, thereby reducing redundant information and computational resource consumption.
It reduces the bandwidth usage of video compression streams, lowers the consumption of device computing resources and power consumption, improves the smoothness of video switching, and ensures the normal transmission and playback of videos from multiple perspectives.
Smart Images

Figure CN117135358B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video compression technology, specifically to a video encoding method, a video decoding method, and related apparatus. Background Technology
[0002] With the development of video compression and communication technologies, panoramic video has emerged in various fields, bringing people new ways of watching and visual experiences. For example, in scenarios such as sports broadcasts, concerts, and game live streams, viewers are no longer satisfied with a single perspective and hope to watch from multiple angles to enjoy a richer video experience.
[0003] Panoramic video supports free switching from 0° to 360° and typically requires the transmission of video from multiple perspectives, each of which is independently encoded and compressed. The simultaneous encoding, compression, transmission, and decoding of multiple perspectives not only consumes significant computing resources and increases power consumption during encoding and decoding, but also occupies excessive channel bandwidth during transmission. In some cases, excessive bandwidth consumption can even prevent transmission, affecting normal video playback. Summary of the Invention
[0004] This application provides a video encoding method, a video decoding method, and related apparatus to reduce resource consumption during the video encoding and decoding process, and to reduce the bandwidth occupied by the video during transmission.
[0005] In a first aspect, embodiments of this application provide a video encoding method, which may include the following steps: acquiring video images from multiple perspectives at the same time, and dividing each video image into N image blocks; generating encoded data for the N image blocks based on the video images from multiple perspectives; and obtaining a compressed video stream from multiple perspectives based on the encoded data of the N image blocks. Wherein, N is an integer greater than 1, and the encoded data of the N image blocks includes encoded data for at least one target image block. The encoded data of the target image block includes encoding information for the target image block from an independent encoding perspective and reference information for a non-independent encoding perspective. The reference information for the non-independent encoding perspective is used to instruct the target image block from the non-independent encoding perspective to decode by referring to the encoding information of the target image block from the independent encoding perspective; both the independent encoding perspective and the non-independent encoding perspective are perspectives among the aforementioned multiple perspectives.
[0006] In the encoded data of N image blocks generated in this embodiment, the encoded data of at least one image block includes reference information for a non-independent encoding viewpoint. This reference information is merely an indication that the image block from that non-independent encoding viewpoint can be decoded by referring to the encoding information of image blocks from other viewpoints. Compared to the encoded data obtained by encoding the image blocks, the amount of reference information is very small. When an image block from a certain non-independent encoding viewpoint can refer to image blocks at the same position in other viewpoints, using reference information can reduce redundant information, improve data compression rate, and reduce the amount of encoded data obtained. This reduces the bandwidth occupied by the video compressed stream, which is beneficial for the normal transmission of video compressed streams from multiple viewpoints and makes video switching between different viewpoints smoother.
[0007] Meanwhile, compared with video images from multiple perspectives being encoded and compressed independently, the embodiments of this application do not perform encoding calculations on image blocks from non-independent encoding perspectives that can refer to other perspectives, which can save the computing resources consumed by the video encoding and decoding process and reduce the power consumption of the device.
[0008] In one possible implementation, each video image can be divided into N image blocks using the same partitioning method, with image blocks at the same location in each video image having the same number. When generating the encoded data for the N image blocks, each numbered image block can be used sequentially as the image block to be encoded, and the encoded data for the image block to be encoded can be generated as follows: The image block to be encoded for the first viewpoint among multiple viewpoints is encoded to obtain the encoded information for the first viewpoint; for each non-first viewpoint among the multiple viewpoints other than the first viewpoint, if the non-first viewpoint is a non-independent encoding viewpoint, reference information for the non-first viewpoint is generated; if the non-first viewpoint is an independent encoding viewpoint, encoded information for the image block to be encoded for the non-first viewpoint is generated; wherein, the first viewpoint is an independent encoding viewpoint.
[0009] The above embodiments process image blocks according to their numbering order, processing all viewpoints with the same number before processing image blocks with the next number. When processing image blocks with the same number, for non-independent coded viewpoints, generating reference information for those viewpoints reduces redundant information between image blocks at the same location across viewpoints.
[0010] In one possible implementation, for a non-first viewpoint, the image block to be encoded from the non-first viewpoint can be compared with the image blocks to be encoded from each of the determined independent encoding viewpoints to determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint. If there is no independent encoding viewpoint whose degree of difference meets the set degree of difference threshold, then the non-first viewpoint is determined as an independent encoding viewpoint, and the image block to be encoded from the non-first viewpoint is encoded to obtain the encoded information of the image block to be encoded from the non-first viewpoint. If there is an independent encoding viewpoint whose degree of difference meets the set degree of difference threshold, then the non-first viewpoint is determined as a non-independent encoding viewpoint, and reference information for the non-first viewpoint is generated based on the information of the candidate independent encoding viewpoints; wherein, the candidate independent encoding viewpoint refers to the independent encoding viewpoint whose degree of difference meets the set degree of difference threshold.
[0011] For example, if the candidate independent coding perspective includes only one independent coding perspective, then the candidate independent coding perspective is taken as the target independent coding perspective, and reference information for the non-first perspective is generated based on the perspective identification information of the target independent coding perspective. If the candidate independent coding perspective includes multiple independent coding perspectives, then the independent coding perspective with the smallest difference is taken as the target independent coding perspective, and reference information for the non-first perspective is generated based on the perspective identification information of the target independent coding perspective.
[0012] In the above embodiments, the reference information of the non-independent coding perspective only contains the perspective identification information of the target independent coding perspective. Its data volume is very small, which can significantly reduce the bandwidth occupied by the transmission coding data and the storage space occupied by the storage coding data.
[0013] In one possible implementation, when generating reference information for a non-first viewpoint, the pixel offset of the image block to be encoded from the non-first viewpoint relative to the image block to be encoded from the target independent encoding viewpoint can be determined, and reference information containing viewpoint identification information and pixel offset from the target independent encoding viewpoint can be generated. The obtained reference information is then used as the reference information for the non-first viewpoint.
[0014] In the above embodiments, the reference information of the non-independent coding perspective includes pixel offsets. During decoding, the image block of the target independent coding perspective can be adjusted based on the pixel offsets to obtain the image block of the non-independent coding perspective more accurately.
[0015] Secondly, embodiments of this application provide a video decoding method, which includes the following steps:
[0016] The system receives a video compressed stream from multiple viewpoints and obtains the encoded data of N image blocks contained in the video compressed stream; where N is an integer greater than 1. The encoded data of the N image blocks is obtained by encoding video images from multiple viewpoints at the same time; the encoded data of the N image blocks includes the encoded data of at least one target image block; the encoded data of the target image block includes the encoding information of the target image block of the independent encoded viewpoint and the reference information of the non-independent encoded viewpoint; the reference information of the non-independent encoded viewpoint is used to instruct the target image block of the non-independent encoded viewpoint to be decoded with reference to the encoding information of the target image block of the independent encoded viewpoint; both the independent encoded viewpoint and the non-independent encoded viewpoint are contained in multiple viewpoints;
[0017] The encoded data of N image blocks is decoded to obtain the decoded data of N image blocks; the decoded data of each image block includes decoded data from multiple viewpoints;
[0018] The decoded data from each viewpoint are stitched together to form a video image, resulting in video images from multiple viewpoints.
[0019] Compared to independently encoding and compressing video images from multiple perspectives, the embodiments of this application do not require independent decoding of reference information for non-independent encoded perspectives contained in the encoded data. Instead, the decoding data of the image block of the corresponding target independent encoded perspective can be obtained by referring to the decoding data of the image block of the non-independent encoded perspective. Therefore, the amount of computation in the decoding process can be reduced, the computing resources consumed in the video decoding process can be saved, and the power consumption of the video playback device can be reduced.
[0020] In one possible implementation, when decoding the encoded data of N image blocks, each image block can be sequentially treated as the image block to be decoded according to its numbering order. The decoded data of the image block to be decoded is obtained as follows: the encoded information of the independent encoded viewpoint contained in the encoded data of the image block to be decoded is decoded to obtain the decoded data of the independent encoded viewpoint; if the encoded data of the image block to be decoded contains reference information of the non-independent encoded viewpoint, the target independent encoded viewpoint referenced by the non-independent encoded viewpoint is determined based on the viewpoint identification information contained in the reference information of the non-independent encoded viewpoint; the decoded data of the non-independent encoded viewpoint is obtained based on the decoded data of the target independent encoded viewpoint; the decoded data of the target independent encoded viewpoint is obtained by decoding the encoded information of the image block to be decoded of the target independent encoded viewpoint.
[0021] In one possible implementation, the reference information of the non-independent coding perspective includes pixel offsets. Based on these pixel offsets, the decoding data of the target independent coding perspective can be adjusted to obtain the decoding data of the non-independent coding perspective.
[0022] Thirdly, embodiments of this application also provide a video encoding apparatus, including a processor and a transceiver;
[0023] The processor is used to: acquire video images from multiple perspectives at the same time, and divide each video image into N image blocks; generate encoded data for N image blocks based on the video images from multiple perspectives, and obtain a compressed video stream from multiple perspectives based on the encoded data of the N image blocks; wherein, N is an integer greater than 1; the encoded data of the N image blocks includes encoded data of at least one target image block; the encoded data of the target image block includes encoded information of the target image block of the independent encoded perspective and reference information of the non-independent encoded perspective; the reference information of the non-independent encoded perspective is used to instruct the target image block of the non-independent encoded perspective to be decoded with reference to the encoded information of the target image block of the independent encoded perspective; both the independent encoded perspective and the non-independent encoded perspective are included in the multiple perspectives.
[0024] A transceiver is used to send compressed video streams from multiple perspectives under the control of a processor.
[0025] In one possible implementation, image blocks at the same location in each video image are numbered the same. Specifically, the processor can be used to sequentially treat each numbered image block as an image block to be encoded according to the numbering order of the N image blocks, and generate the encoded data of the image block to be encoded in the following way: Encode the image block to be encoded in the first view among multiple viewpoints to obtain the encoded information of the first viewpoint; wherein, the first viewpoint is an independent encoded viewpoint; for each non-first viewpoint among multiple viewpoints other than the first viewpoint, if the non-first viewpoint is a non-independent encoded viewpoint, then generate reference information for the non-first viewpoint; if the non-first viewpoint is an independent encoded viewpoint, then generate the encoded information of the image block to be encoded in the non-first viewpoint.
[0026] In one possible implementation, the processor can specifically be used to: compare the image blocks to be encoded from non-first viewpoints with the image blocks to be encoded from each determined independent encoding viewpoint, and determine the degree of difference between each independent encoding viewpoint and the non-first viewpoints; if there is an independent encoding viewpoint whose degree of difference meets a set degree of difference threshold, then determine the non-first viewpoint as a non-independent encoding viewpoint; generate reference information for the non-first viewpoint based on the information of the candidate independent encoding viewpoints, wherein the candidate independent encoding viewpoints refer to independent encoding viewpoints whose degree of difference meets a set degree of difference threshold.
[0027] In one possible implementation, the processor can specifically be used to: if a candidate independent coding perspective includes an independent coding perspective, then use the candidate independent coding perspective as the target independent coding perspective; and generate reference information for a non-first perspective based on the perspective identification information of the target independent coding perspective.
[0028] In one possible implementation, the processor can be used to: if the candidate independent coding perspective includes multiple independent coding perspectives, then select the independent coding perspective with the smallest difference as the target independent coding perspective; and generate reference information for non-first perspectives based on the perspective identification information of the target independent coding perspective.
[0029] In one possible implementation, the processor can specifically be used to: determine the pixel offset of the image block to be encoded from a non-first viewpoint relative to the image block to be encoded from the target independent encoding viewpoint; and generate reference information containing viewpoint identification information and pixel offsets of the target independent encoding viewpoint, wherein the reference information is reference information for the non-first viewpoint.
[0030] In one possible implementation, the processor can specifically be used to: compare the image blocks to be encoded from non-first viewpoints with the image blocks to be encoded from each determined independent encoding viewpoint, and determine the degree of difference between each independent encoding viewpoint and the non-first viewpoints; if there is no independent encoding viewpoint whose degree of difference meets the set degree of difference threshold, then determine the non-first viewpoint as an independent encoding viewpoint; encode the image blocks to be encoded from the non-first viewpoints to obtain the encoding information of the image blocks to be encoded from the non-first viewpoints.
[0031] Fourthly, embodiments of this application also provide a video decoding device, including a transceiver and a processor;
[0032] The transceiver is used to receive compressed video streams from multiple perspectives under the control of the processor;
[0033] The processor is configured to: acquire encoded data of N image blocks contained in the video compressed stream, where N is an integer greater than 1; the encoded data of the N image blocks is obtained by encoding video images from multiple perspectives at the same time; the encoded data of the N image blocks includes encoded data of at least one target image block; the encoded data of the target image block includes encoding information of the target image block from an independent encoding perspective and reference information of a non-independent encoding perspective; the reference information of the non-independent encoding perspective is used to instruct the target image block from the non-independent encoding perspective to be decoded by referring to the encoding information of the target image block from an independent encoding perspective; both independent and non-independent encoding perspectives are contained in multiple perspectives;
[0034] Decode the encoded data of N image blocks to obtain decoded data of N image blocks. The decoded data of each image block includes decoded data from multiple viewpoints.
[0035] The decoded data from each viewpoint are stitched together to form a video image, resulting in video images from multiple viewpoints.
[0036] In one possible implementation, the processor can specifically be used to: sequentially treat each numbered image block as an image block to be decoded according to the numbering order of N image blocks, and obtain the decoding data of the image block to be decoded in the following way: decode the encoding information of the independent coded viewpoint contained in the encoding data of the image block to be decoded to obtain the decoding data of the independent coded viewpoint; if the encoding data of the image block to be decoded contains reference information of the non-independent coded viewpoint, then determine the target independent coded viewpoint referenced by the non-independent coded viewpoint based on the viewpoint identification information contained in the reference information of the non-independent coded viewpoint; obtain the decoding data of the non-independent coded viewpoint based on the decoding data of the target independent coded viewpoint; the decoding data of the target independent coded viewpoint is obtained by decoding the encoding information of the image block to be decoded of the target independent coded viewpoint.
[0037] In one possible implementation, the reference information for the non-independent coding view includes pixel offsets. Specifically, the processor can be used to: adjust the decoded data of the target independent coding view based on the pixel offsets to obtain the decoded data of the non-independent coding view.
[0038] Fifthly, embodiments of this application also provide a video encoding apparatus, which includes corresponding functional modules, each used to implement the steps in the methods provided in the first aspect. For details, please refer to the detailed description in the method examples; further details are omitted here. Functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the video encoding apparatus may include an image partitioning unit and an encoding unit. The image partitioning unit is used to acquire video images from multiple perspectives at the same time and divide each video image into N image blocks; where N is an integer greater than 1. The encoding unit is used to generate encoded data for the N image blocks based on the video images from multiple perspectives, and to obtain a video compressed stream from multiple perspectives based on the encoded data of the N image blocks; wherein the encoded data of the N image blocks includes encoded data of at least one target image block; the encoded data of the target image block includes encoding information of the target image block from an independent encoding perspective and reference information of a non-independent encoding perspective; the reference information of the non-independent encoding perspective is used to instruct the target image block of the non-independent encoding perspective to decode by referring to the encoding information of the target image block from an independent encoding perspective; both independent and non-independent encoding perspectives are included in multiple perspectives.
[0039] Sixthly, embodiments of this application also provide a video decoding apparatus, which includes corresponding functional modules, each used to implement the steps in the methods provided in the second aspect. For details, please refer to the detailed description in the method examples, which will not be repeated here. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, a video decoding device may include a data receiving unit and a decoding unit; the data receiving unit is used to receive a video compressed stream from multiple perspectives and acquire the encoded data of N image blocks contained in the video compressed stream, where N is an integer greater than 1; the encoded data of the N image blocks is obtained by encoding video images from multiple perspectives at the same time; the encoded data of the N image blocks includes the encoded data of at least one target image block; the encoded data of the target image block includes the encoding information of the target image block of the independent encoded perspective and the reference information of the non-independent encoded perspective; the reference information of the non-independent encoded perspective is used to instruct the target image block of the non-independent encoded perspective to be decoded with reference to the encoding information of the target image block of the independent encoded perspective; both the independent encoded perspective and the non-independent encoded perspective are included in multiple perspectives; the decoding unit is used to decode the encoded data of the N image blocks to obtain the decoded data of the N image blocks, and the decoded data of each image block includes the decoded data of multiple perspectives; the decoded data of each perspective are stitched together to form a video image to obtain a video image of multiple perspectives.
[0040] In a seventh aspect, embodiments of this application also provide a video transmission system, including any of the video encoding devices provided in the third aspect and any of the video decoding devices provided in the fourth aspect.
[0041] Eighthly, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed by a terminal device, cause the terminal device to perform any of the methods provided in the first aspect or any of the methods provided in the second aspect.
[0042] Ninthly, this application provides a computer program product comprising a computer program or instructions that, when executed by a terminal device, implement any of the methods provided in the first aspect or any of the methods provided in the second aspect.
[0043] The technical effects that can be achieved by any of the third to ninth aspects mentioned above can be referred to the description of the beneficial effects in the first or second aspects mentioned above, and will not be repeated here. Attached Figure Description
[0044] Figure 1 This is a schematic diagram of the structure of a video transmission system provided in an embodiment of this application;
[0045] Figure 2 This is a schematic diagram of a video encoding and decoding process using related technologies.
[0046] Figure 3 A flowchart illustrating a video encoding method provided in this application embodiment;
[0047] Figure 4 A flowchart illustrating another video encoding method provided in this application embodiment;
[0048] Figure 5 A schematic diagram of video images from multiple perspectives provided in embodiments of this application;
[0049] Figure 6 A schematic diagram of a video compression stream provided in an embodiment of this application;
[0050] Figure 7 A flowchart illustrating a video decoding method provided in this application embodiment;
[0051] Figure 8 A schematic diagram illustrating a video encoding / decoding process provided in an embodiment of this application;
[0052] Figure 9 A schematic diagram of a video encoding device provided in an embodiment of this application;
[0053] Figure 10 This is a schematic diagram of a video decoding device provided in an embodiment of this application. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only used to explain specific embodiments of this application and is not intended to limit this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0055] Before introducing the specific solutions provided in the embodiments of this application, some terms used in this application will be explained in a general way to facilitate understanding by those skilled in the art, and the terms used in this application will not be limited.
[0056] (1) Perspective: In the application scenarios of virtual reality (VR) video, augmented reality (AR) video or 360° panoramic video, the video content that a user sees at a certain moment is called a perspective.
[0057] In this application embodiment, "multiple" refers to two or more. Therefore, in this application embodiment, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, such as one, two, or more. For example, "including at least one" means including one, two, or more, and it does not limit which ones are included. For example, including at least one of A, B, and C, then it could include A, B, C, A and B, A and C, B and C, or A and B and C. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.
[0058] Unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects, and are not used to limit the order, sequence, priority or importance of multiple objects.
[0059] Video compression technology is widely used in the Internet, television broadcasting, storage media, communications, and other fields. Video compression technology consists of two parts: video encoding and video decoding. Video encoding generally refers to processing the sequence of images that form a video. Video encoding is performed on the source device side and typically involves compressing the raw video images to reduce the amount of data required to represent them, thus enabling more efficient storage and transmission. Video decoding is performed on the destination device side and typically involves inverse processing relative to the encoder to reconstruct the video images. In the field of video compression technology, the terms "picture," "frame," or "image" can be used synonymously.
[0060] Figure 1 An exemplary application scenario of video compression technology is shown, which can be viewed as a video transmission system. For example... Figure 1 As shown, the video transmission system includes a video source device 100 and a video playback device 200, wherein the video playback device can be understood as the destination device.
[0061] The video source device 100 is used to acquire video, encode the video, and provide the encoded data to the video playback device 200. The video source device 100 may include or be any type of image capture device for capturing real-world images, or any type of image generation device. For example, the video source device 100 may include a computer graphics processor for generating computer-animated images, or the video source device 100 may include any type of device for acquiring real-world images, providing computer-generated VR images, or providing AR images that combine real-world images with virtual images.
[0062] In some embodiments, the video source device 100 can directly send the encoded data to the video playback device 200 after encoding the video. For example, the video source device 100 and the video playback device 200 can communicate via wired or wireless communication. In other embodiments, the video source device 100 can save the encoded data to a storage device after encoding the video, and the video playback device 200 can then retrieve the encoded data from the storage device.
[0063] The video playback device 200 is used to decode the acquired encoded data and output the decoded video image to the display device for playback.
[0064] With the development of video compression and communication technologies, VR, AR, and 360° panoramic videos have emerged in various fields, bringing people new ways of watching and visual experiences. For example, in scenarios such as sports broadcasts, concerts, and game live streams, viewers are no longer satisfied with a single perspective and hope to watch from multiple angles to enjoy a richer video experience.
[0065] Panoramic video supports free switching from 0 to 360 degrees, allowing users to freely switch viewing angles across the entire screen on the video playback device. To achieve the best viewing experience, multiple viewing angles are typically transmitted, with each angle's video independently encoded and compressed. For example, such as... Figure 2 As shown, suppose we need to transmit video images from four perspectives: perspective 0, perspective 1, perspective 2, and perspective 3. In the video source device, each of the four perspective video images needs to be encoded using an independent video encoding module. The encoded video data is then transmitted to the video playback device. In the video playback device, the encoded video data from the four perspectives needs to be decoded using an independent video decoding module to obtain the four perspective video images.
[0066] Encoding, compressing, transmitting, and decoding videos from multiple perspectives simultaneously not only consumes significant computing resources from both the video source and playback devices during encoding and decoding, increasing power consumption, but also occupies excessive channel bandwidth during transmission. In some cases, excessive bandwidth consumption can even prevent transmission, affecting normal video playback. For example, if a user needs to switch from one perspective to another before the desired video image arrives, stuttering will occur, impacting the user's viewing experience.
[0067] For example, taking a television as the video playback device 200, current high-definition televisions can play 4K or 8K videos. The video source device 11 needs a bandwidth of more than 25Mbps to transmit one 4K video viewpoint to the video playback device 12. Assuming that panoramic video requires at least 10 4K video viewpoints, the bandwidth required to transmit 10 4K video viewpoints is more than 250Mbps. Most broadband cannot meet this bandwidth requirement.
[0068] Based on this, embodiments of this application provide a video encoding and decoding method, wherein the video encoding method can be derived from... Figure 1 The video source device 100 in the middle executes the video decoding method, which can be performed by... Figure 1 The video playback device 200 executes the following. The video encoding method includes: a video source device acquiring video images from multiple perspectives and dividing each video image into N image blocks, where N is an integer greater than 1; the video source device generating encoded data for the N image blocks based on the video images from multiple perspectives, and obtaining a compressed video stream from multiple perspectives based on the encoded data of the N image blocks. The encoded data of the N image blocks includes encoded data for at least one target image block, and the encoded data of the target image block includes encoding information for the target image block of an independent encoding perspective and reference information for a non-independent encoding perspective. The reference information for a non-independent encoding perspective is used to indicate that the target image block of the non-independent encoding perspective can be decoded by referring to the encoding information of the target image block of the independent encoding perspective, wherein both the independent encoding perspective and the non-independent encoding perspective are perspectives among the aforementioned multiple perspectives.
[0069] In the encoded data of N image blocks generated by the video source device in this application embodiment, the encoded data of at least one image block includes reference information for a non-independent encoding viewpoint. This reference information is merely an indication that the image block from that non-independent encoding viewpoint can be decoded by referring to the encoding information of image blocks from other viewpoints. Compared to the encoded data obtained by encoding the image blocks, the amount of reference information is very small. When an image block from a certain non-independent encoding viewpoint can refer to image blocks at the same position in other viewpoints, using reference information can reduce redundant information, improve data compression rate, and reduce the amount of encoded data obtained. This reduces the bandwidth occupied by the video compressed stream, which is beneficial for the normal transmission of video compressed streams from multiple viewpoints, making video switching between different viewpoints smoother and allowing users to watch videos more smoothly.
[0070] Meanwhile, compared with video images from multiple perspectives being encoded and compressed independently, the embodiments of this application do not perform encoding calculations on image blocks from non-independent encoding perspectives that can refer to other perspectives, which can save the computing resources consumed by the video encoding and decoding process and reduce the power consumption of the device.
[0071] Figure 3An exemplary flowchart of a video encoding method provided in an embodiment of this application is shown. This video encoding method can be... Figure 1 The video source device 100 in the middle executes, such as Figure 3 As shown, the method may include the following steps:
[0072] S301: Acquire video images from multiple perspectives at the same time and divide each video image into N image blocks.
[0073] The video source device captures video images of a real-world scene from multiple perspectives, or generates video images from multiple perspectives. For video images captured or generated simultaneously from multiple perspectives, the video source device divides each perspective video image into N image blocks using the same method. Here, N is an integer greater than 1. Image blocks at the same location in each video image have the same number.
[0074] S302 generates encoded data for N image blocks based on video images from multiple perspectives.
[0075] The video source device can sequentially use each of the N image blocks as the image block to be encoded, according to their numbering order, and generate the encoded data of the image block to be encoded in the following way:
[0076] Encode the image block to be encoded in the first view of multiple viewpoints to obtain the encoding information of the first viewpoint, where the first viewpoint is an independent encoding viewpoint;
[0077] For each non-first viewpoint among multiple viewpoints (excluding the first viewpoint), the following operations can be performed: The image block to be encoded from the non-first viewpoint is compared with the image blocks to be encoded from each of the determined independent encoding viewpoints. The difference degree of each independent encoding viewpoint relative to the non-first viewpoint is determined. If no independent encoding viewpoint has a difference degree that meets the set difference degree threshold, the non-first viewpoint is determined as an independent encoding viewpoint, and its image block to be encoded is encoded to obtain the encoded information of the non-first viewpoint image block. If an independent encoding viewpoint has a difference degree that meets the set difference degree threshold, the non-first viewpoint is determined as a non-independent encoding viewpoint. Reference information for the non-first viewpoint is generated based on the information of candidate independent encoding viewpoints, where candidate independent encoding viewpoints refer to independent encoding viewpoints whose difference degree meets the set difference degree threshold.
[0078] For example, if a candidate independent coding viewpoint includes only one independent coding viewpoint, then that candidate independent coding viewpoint is taken as the target independent coding viewpoint, and reference information for the non-first viewpoint is generated based on the viewpoint identification information of the target independent coding viewpoint. If a candidate independent coding viewpoint includes multiple independent coding viewpoints, then the independent coding viewpoint with the smallest difference is taken as the target independent coding viewpoint, and reference information for the non-first viewpoint is generated based on the viewpoint identification information of the target independent coding viewpoint. The reference information for the non-first viewpoint includes the viewpoint identification information of the target independent coding viewpoint, which is used to indicate that the image block to be encoded in the non-first viewpoint can be decoded by referring to the encoding information of the image block to be encoded in the target independent coding viewpoint. Through the above process, the encoded data of the image block to be encoded can be obtained. For the image block to be encoded, if multiple views are all independent coding viewpoints, then the encoded data of the image block to be encoded includes the encoding information of the image block to be encoded in each independent coding viewpoint; if multiple views include both independent coding viewpoints and non-independent coding viewpoints, then the encoded data of the image block to be encoded includes the encoding information of the image block to be encoded in each independent coding viewpoint and the reference information of each non-independent coding viewpoint.
[0079] Following the above process, the encoded data for each numbered image block can be obtained, resulting in encoded data for N image blocks. Among the encoded data of these N image blocks, at least one target image block's encoded data is included. A target image block refers to an image block whose encoded data includes both encoded information from an independent encoded viewpoint and reference information from a non-independent encoded viewpoint. The reference information from the non-independent encoded viewpoint indicates that the target image block from that non-independent encoded viewpoint can be decoded by referring to the encoded information of the target image block from the independent encoded viewpoint.
[0080] S303, based on the encoded data of N image blocks, obtains a video compressed stream from multiple perspectives.
[0081] Based on the encoded data of the obtained N image blocks, a video compressed stream from the above multiple perspectives is generated, and the video compressed stream from the multiple perspectives is transmitted to the video playback device to provide panoramic video for the video playback device.
[0082] In the above embodiments, the video compression stream from multiple perspectives includes reference information. The amount of reference information is very small. When an image block from a non-independent coded perspective can refer to image blocks at the same position from other perspectives, using reference information can reduce redundant information, improve the data compression rate, and reduce the amount of data in the video compression stream, thereby reducing the bandwidth occupied by the video compression stream.
[0083] For ease of understanding, the video encoding method provided in this application embodiment will be described in detail below through a specific example. For example... Figure 4 As shown, the video encoding method may include the following steps:
[0084] S401: Acquire video images from M perspectives at the same time and divide each video image into N image blocks.
[0085] This application embodiment encodes based on image blocks. It assumes that a video source device provides video images from M viewpoints to a video playback device. After acquiring the video images from the M viewpoints at the same time, the video source device divides each viewpoint's video image into N image blocks using the same partitioning method. Image blocks can also be called coding tree blocks (CTBs), coding tree units (CTUs), or coding units (CUs). For example, assuming M is 4, the four viewpoints are viewpoint 0, viewpoint 1, viewpoint 2, and viewpoint 3; and N is 6, the video images from the four viewpoints are divided into 6 image blocks. Figure 5 As shown, the video image at viewpoint 0 is divided into 6 image blocks, namely CTB0, CTB1, CTB2, CTB3, CTB4 and CTB5. The video images at viewpoints 1, 2 and 3 are also divided in the same way, and the image blocks at the same position in each video image are numbered the same.
[0086] S402, select the image block with the smallest number as the image block to be encoded.
[0087] The video source device can generate the encoded data of each image block in ascending or descending order of the numbering of N image blocks. In this embodiment, the image block with the smallest number, i.e., the image block with the smallest number, i.e., the image block with the smallest number, is taken as the image block to be encoded. Then, in ascending order of the numbers, each image block with the smallest number is taken as the image block to be encoded. The image blocks to be encoded are processed according to steps S403 to S407.
[0088] S403, Encode the image block to be encoded at viewpoint 0 to obtain the encoded information of the image block to be encoded at viewpoint 0.
[0089] Taking image block numbered 0 (CTB0) as an example, the CTB0 of viewpoint 0 is encoded. For instance, the CTB0 of viewpoint 0 can be encoded according to video compression protocols such as H.264 and High Efficiency Video Coding (HEVC) introduced by international standards organizations. CTB0 is divided into multiple pixel blocks, and the residuals and motion vector differences (MVDs) between each pixel block and a set reference frame or reference pixel block are determined. Encoded data containing the identification information of viewpoint 0 of CTB0 and the residuals and MVDs of each pixel block of CTB0 is generated. This encoded data is the encoding information of viewpoint 0 of CTB0. Since viewpoint 0 is the first viewpoint, it cannot refer to the encoding information of image blocks from other viewpoints; therefore, viewpoint 0 is an independently encoded viewpoint.
[0090] S404, determine whether the image block numbered 0 of the next viewpoint can use the encoding information of the image block numbered 0 of any determined independent encoding viewpoint; if yes, proceed to step S406, if no, proceed to step S405.
[0091] S405, determine that for the image block to be encoded, the next viewpoint is an independent encoding viewpoint, encode the image block numbered 0 of the next viewpoint, and obtain the encoding information of the image block numbered 0 of the next viewpoint.
[0092] S406, determine that for the image block to be encoded, the next viewpoint is a non-independent coding viewpoint, and generate reference information for the next viewpoint based on the information of the available independent coding viewpoints.
[0093] For example, the next viewpoint after viewpoint 0 is viewpoint 1. The CTB0 of viewpoint 1 is compared with the CTB0 of viewpoint 0. By comparing the pixel value differences between corresponding pixels in the CTB0 of viewpoint 1 and the CTB0 of viewpoint 0, the degree of difference between the CTB0 of viewpoint 1 and the CTB0 of viewpoint 0 is determined. If the degree of difference between the CTB0 of viewpoint 1 and the CTB0 of viewpoint 0 is greater than a set degree of difference threshold, it indicates that the degree of difference between the CTB0 of viewpoint 1 and the CTB0 of viewpoint 0 is large. Viewpoint 1's CTB0 cannot use the encoding information of viewpoint 0's CTB0. Therefore, it is determined that viewpoint 1 is an independent encoding viewpoint for CTB0, and the CTB0 of viewpoint 1 is encoded to obtain the encoding information of viewpoint 1's CTB0. For example, the degree of difference threshold can be set to 20%. The specific process of encoding the CTB0 of viewpoint 1 can be performed in the same way as the process of encoding the CTB0 of viewpoint 0, and will not be repeated here. If the difference between CTB0 of viewpoint 1 and CTB0 of viewpoint 0 is less than or equal to the set difference threshold, it means that the difference between CTB0 of viewpoint 1 and CTB0 of viewpoint 0 is small. CTB0 of viewpoint 1 can use the encoding information of CTB0 of viewpoint 0. Therefore, it is determined that viewpoint 1 is a non-independent encoded viewpoint for CTB0, and reference information of viewpoint 1 can be generated based on the information of viewpoint 0.
[0094] In one embodiment, if it is determined that the CTB0 of viewpoint 1 can use the encoding information of the CTB0 of viewpoint 0, then reference information of viewpoint 1 can be generated based on the viewpoint identification information of viewpoint 0, that is, reference information containing the viewpoint identification information of viewpoint 0 can be generated as reference information of viewpoint 1. This reference information is used to indicate that the CTB0 of viewpoint 1 can be decoded by referring to the encoding information of the CTB0 of viewpoint 0.
[0095] In another embodiment, the reference information for viewpoint 1 may include, in addition to the viewpoint identification information of viewpoint 0, the pixel offset of CTB0 of viewpoint 1 relative to CTB0 of viewpoint 0. That is, if it is determined that CTB0 of viewpoint 1 can use the encoding information of CTB0 of viewpoint 0, the pixel offset of CTB0 of viewpoint 1 relative to CTB0 of viewpoint 0 can be determined based on CTB0 of viewpoint 1 and CTB0 of viewpoint 0. For example, the pixel offset of CTB0 of viewpoint 1 relative to CTB0 of viewpoint 0 can be determined by comparing the position of the same object in CTB0 of viewpoint 1 with its position in CTB0 of viewpoint 0. The pixel offset may include one or more of the following: horizontal offset, vertical offset, or rotation angle offset of a pixel. After determining the pixel offset of CTB0 of viewpoint 1 relative to CTB0 of viewpoint 0, reference information containing the viewpoint identification information of viewpoint 0 and the pixel offset can be generated as reference information for viewpoint 1. During image decoding, the CTB0 of the decoded viewpoint 0 can be adjusted according to the pixel offset in the reference information to obtain the CTB0 of the viewpoint 1.
[0096] In some embodiments, the generated reference information or encoded information of viewpoint 1 includes a 1-bit flag bit, which is used to identify whether the corresponding information is reference information or encoded information. For example, if the flag bit is "0", it indicates that the corresponding information is reference information of viewpoint 1; that is, if viewpoint 1 is determined to be a non-independent encoded viewpoint, the flag bit is set to "0". If the flag bit is "1", it indicates that the corresponding information is encoded information of viewpoint 1; that is, if viewpoint 1 is determined to be an independent encoded viewpoint, the flag bit is set to "1".
[0097] S407, determine if this is the last perspective; if not, return to step S404; if yes, proceed to step S408.
[0098] For example, if viewpoint 1 is not the last viewpoint, and the next viewpoint after viewpoint 1 is viewpoint 2, then return to step S404 and determine whether the CTB0 of viewpoint 2 can use the encoding information of the CTB0 of any determined independent encoding viewpoint.
[0099] In some embodiments, if viewpoint 0 and viewpoint 1 are both independently coded viewpoints for CTB0, then the CTB0 of viewpoint 2 is compared with the CTB0 of viewpoint 0 and the CTB0 of viewpoint 1 respectively to determine the degree of difference between the CTB0 of viewpoint 0 and the CTB0 of viewpoint 2, and the degree of difference between the CTB0 of viewpoint 1 and the CTB0 of viewpoint 2. The degree of difference between the CTB0 of viewpoint 0 and the CTB0 of viewpoint 2 is called the degree of difference corresponding to viewpoint 0, and the degree of difference between the CTB0 of viewpoint 1 and the CTB0 of viewpoint 2 is called the degree of difference corresponding to viewpoint 1.
[0100] If the difference degree corresponding to viewpoint 0 and viewpoint 1 are both greater than the set difference degree threshold, it means that the difference degree between viewpoint 2's CTB0 and both viewpoint 0's and viewpoint 1's CTB0 is large. Viewpoint 2's CTB0 cannot use the encoding information of viewpoint 0's CTB0, nor can it use the encoding information of viewpoint 1's CTB0. Therefore, viewpoint 2 is determined to be an independently encoded viewpoint for CTB0, and its CTB0 is encoded to obtain the encoded information. The specific process of encoding viewpoint 2's CTB0 can be referred to the process of encoding viewpoint 0's CTB0, and will not be repeated here.
[0101] If the difference degree corresponding to viewpoint 0 is less than or equal to the set difference degree threshold, and the difference degree corresponding to viewpoint 1 is greater than the set difference degree threshold, it means that the difference degree between CTB0 of viewpoint 2 and CTB0 of viewpoint 0 is small. CTB0 of viewpoint 2 can use the encoding information of CTB0 of viewpoint 0. Therefore, it is determined that for CTB0, viewpoint 2 is a non-independent encoded viewpoint, and reference information of viewpoint 2 can be generated based on the information of viewpoint 0.
[0102] If the difference degree corresponding to viewpoint 0 is greater than the set difference degree threshold, and the difference degree corresponding to viewpoint 1 is less than or equal to the set difference degree threshold, it means that the difference degree between CTB0 of viewpoint 2 and CTB0 of viewpoint 1 is small. CTB0 of viewpoint 2 can use the encoding information of CTB0 of viewpoint 1. Therefore, it is determined that for CTB0, viewpoint 2 is a non-independent encoded viewpoint, and reference information of viewpoint 2 can be generated based on the information of viewpoint 1.
[0103] If the difference degree corresponding to viewpoint 0 and the difference degree corresponding to viewpoint 1 are both less than or equal to the set difference degree threshold, it means that the difference degree between viewpoint 2's CTB0 and both viewpoint 0's and viewpoint 1's CTB0 is small. Viewpoint 2's CTB0 can use the encoding information of viewpoint 0's CTB0 or viewpoint 1's CTB0. Both viewpoint 0 and viewpoint 1 can be used as candidate independent encoding viewpoints. At this time, the difference degree corresponding to viewpoint 0 and viewpoint 1 can be compared. The viewpoint with the smaller difference degree is selected as the target independent encoding viewpoint. Based on the information of the target independent encoding viewpoint, reference information for this non-first viewpoint is generated. For example, if the difference degree corresponding to viewpoint 0 is less than the difference degree corresponding to viewpoint 1, then reference information for viewpoint 2 is generated based on the information of viewpoint 0; if the difference degree corresponding to viewpoint 0 is greater than the difference degree corresponding to viewpoint 1, then reference information for viewpoint 2 is generated based on the information of viewpoint 1.
[0104] In other embodiments, if viewpoint 0 is an independently coded viewpoint and viewpoint 1 is a non-independently coded viewpoint for CTB0, then the CTB0 of viewpoint 2 is compared with the CTB0 of viewpoint 0 to determine the degree of difference between the CTB0 of viewpoint 0 and the CTB0 of viewpoint 2, thereby determining whether viewpoint 2 is an independently coded viewpoint for CTB0. If viewpoint 2 is an independently coded viewpoint for CTB0, then the CTB0 of viewpoint 2 is encoded to obtain the encoded information of the CTB0 of viewpoint 2; if viewpoint 2 is a non-independently coded viewpoint for CTB0, then reference information for viewpoint 2 is generated based on the information of viewpoint 0.
[0105] After processing CTB0 of viewpoint 2, since viewpoint 2 is not the last viewpoint, and the next viewpoint is viewpoint 3, we return to step S404 and process CTB0 of viewpoint 3 according to the above process. Since viewpoint 3 is the last viewpoint, after processing CTB0 of viewpoint 3, we continue to execute step S408.
[0106] S408, determine if it is the last numbered image block; if not, proceed to step S409; if yes, proceed to step S410.
[0107] S409, according to the numbering order, take the next numbered image block as the image block to be encoded; return to step S403.
[0108] After processing CTB0 for all viewpoints and generating CTB0 encoded data, the above process can be used to process CTB1 to CTB5 sequentially to obtain CTB1 to CTB5 encoded data.
[0109] S410, based on the encoded data of N image blocks, obtains a video compressed stream with M perspectives.
[0110] The video source device obtains a compressed video stream with four perspectives based on the encoded data of six image blocks (CTB0 to CTB5), and transmits the compressed video stream with four perspectives to the video playback device. The structure of the compressed video stream with four perspectives obtained based on the encoded data of the six image blocks (CTB0 to CTB5) is as follows: Figure 6As shown. The header information can include an image header and a slice header. The slice header is the header of the video slice, also known as the slice header or slice head. Following the header information, the CTB0 information for viewpoint 0 represents the encoded CTB0 information for viewpoint 0. If viewpoint 1 is an independently encoded viewpoint for CTB0, then the CTB0 information for viewpoint 1 represents the encoded CTB0 information for viewpoint 1. If viewpoint 1 is a non-independently encoded viewpoint for CTB0, then the CTB0 information for viewpoint 1 represents the reference information for viewpoint 1. The meanings of the CTB information for other viewpoints can be referenced to the meanings of the CTB0 information for viewpoint 1, and will not be elaborated further here.
[0111] After processing the video images from the four perspectives of this moment or this frame, you can continue to process the video images from the four perspectives of the next moment or the next frame, thereby sending a continuous video compressed stream.
[0112] In the above embodiments, dividing the video image into 6 image blocks is merely illustrative. In other embodiments, the video image can be divided into more or fewer image blocks. The 4 perspectives are also illustrative; in actual use, it may be necessary to transmit video from more perspectives or fewer perspectives.
[0113] Compared to independently encoding and compressing video images from multiple perspectives, the video source device in this application embodiment does not perform encoding calculations on image blocks from non-independent encoding perspectives that can refer to other perspectives. This can save computing resources consumed in the video encoding process and reduce the power consumption of the video source device.
[0114] Based on the same inventive concept as the video encoding method in the above embodiments, this application also provides a video decoding method. Figure 7 An exemplary flowchart of a video decoding method provided in an embodiment of this application is shown. This video decoding method can be performed by... Figure 1 The video source device 100 in the middle executes, such as Figure 7 As shown, the method may include the following steps:
[0115] S701 receives video compressed streams from multiple perspectives and obtains the encoded data of N image blocks contained in the video compressed stream.
[0116] The video compressed stream includes coded data for N image blocks, where N is an integer greater than 1. The coded data for these N image blocks is obtained by encoding video images from multiple viewpoints at the same time. The coded data for these N image blocks includes coded data for at least one target image block. The coded data for the target image block includes coded information for the target image block from an independent coded viewpoint and reference information for non-independent coded viewpoints. Both the independent and non-independent coded viewpoints are included in multiple viewpoints. The reference information for the non-independent coded viewpoints is used to instruct the target image blocks from the non-independent coded viewpoints to be decoded by referring to the coded information of the target image blocks from the independent coded viewpoints.
[0117] S702 decodes the encoded data of N image blocks to obtain the decoded data of N image blocks.
[0118] For example, when decoding the encoded data of N image blocks, each numbered image block can be used as the image block to be decoded in the order of their numbers, and the decoded data of the image block to be decoded can be obtained in the following way:
[0119] The encoding information of the independent coded viewpoint contained in the encoded data of the image block to be decoded is decoded to obtain the decoded data of the independent coded viewpoint. If the encoded data of the image block to be decoded contains reference information of a non-independent coded viewpoint, the target independent coded viewpoint referenced by the non-independent coded viewpoint is determined based on the viewpoint identification information contained in the reference information of the non-independent coded viewpoint. Based on the decoded data of the target independent coded viewpoint, the decoded data of the non-independent coded viewpoint is obtained. The decoded data of the target independent coded viewpoint is obtained by decoding the encoded information of the image block to be decoded from the target independent coded viewpoint.
[0120] In some embodiments, a portion of the image blocks to be decoded may contain only encoded information from independent coded viewpoints, i.e., multiple viewpoints are all independently coded viewpoints. In this case, the encoded information from multiple viewpoints contained in the encoded data of the image blocks to be decoded is decoded to obtain decoded data from multiple viewpoints. Another portion of the image blocks to be decoded may contain encoded information from independent coded viewpoints and reference information from non-independent coded viewpoints.
[0121] If the encoded data of the image block to be decoded contains reference information of a non-independent coded viewpoint, the reference information of the non-independent coded viewpoint can be obtained in the following way:
[0122] In one embodiment, if the reference information of the non-independent coding view contains only view identification information, the target independent coding view referenced by the non-independent coding view can be determined based on the view identification information, the decoding data of the target independent coding view can be obtained, and the decoding data of the target independent coding view can be used as the decoding data of the non-independent coding view.
[0123] For example, assuming CTB0 is the image block to be decoded, the encoded data of CTB0 includes reference information for viewpoint 2, and the viewpoint identification information included in the reference information of viewpoint 2 is the identifier of viewpoint 1. Then it can be determined that the target independent encoded viewpoint referenced by viewpoint 2 is viewpoint 1. The decoded data of CTB0 of viewpoint 1 obtained by decoding the encoded information of CTB0 of viewpoint 1 is obtained, and the decoded data of CTB0 of viewpoint 1 is used as the decoded data of CTB0 of viewpoint 2.
[0124] In another embodiment, if the reference information of the non-independent coding view includes viewpoint identification information and pixel offset, the target independent coding view referenced by the non-independent coding view can be determined based on the viewpoint identification information, the decoding data of the target independent coding view can be obtained, and the decoding data of the target independent coding view can be adjusted based on the pixel offset contained in the reference information to obtain the decoding data of the non-independent coding view. The pixel offset can be a horizontal offset, a vertical offset, or a rotation angle offset of a pixel, or it can include any combination of two or three of these offsets.
[0125] For example, assuming CTB0 is the image block to be decoded, and the encoded data of CTB0 includes reference information for viewpoint 2, and the viewpoint identification information included in the reference information of viewpoint 2 is the identifier of viewpoint 0, then it can be determined that the target independent encoded viewpoint referenced by viewpoint 2 is viewpoint 0. Therefore, the decoded data of CTB0 for viewpoint 0 is obtained by decoding the encoded information of CTB0 for viewpoint 0. Assuming that the reference information for viewpoint 2 also includes the horizontal offset of pixels, and the horizontal offset of pixels indicates that CTB0 for viewpoint 2 is shifted d1 columns to the left based on CTB0 for viewpoint 0, then according to the horizontal offset of pixels in the reference information for viewpoint 2, each pixel in the decoded data of CTB0 for viewpoint 0 is shifted d1 columns to the left. After the shift, the pixels in the left d1 column are deleted, and the blank pixels in the right d1 column can be filled by pixels extracted from the decoded data of CTB0 for viewpoint 1.
[0126] For example, assuming CTB3 is the image block to be decoded, the encoded data of CTB3 includes reference information of viewpoint 2, and the viewpoint identification information included in the reference information of viewpoint 2 is the identifier of viewpoint 0. Then it can be determined that the target independent encoded viewpoint referenced by viewpoint 2 is viewpoint 0. Then the decoded data of CTB3 of viewpoint 0 is obtained by decoding the encoded information of CTB3 of viewpoint 0. Assuming that the reference information for viewpoint 2 also includes the horizontal and vertical offsets of pixels, the horizontal offset of pixels indicates that the CTB3 of viewpoint 2 is shifted d1 columns to the left based on the CTB3 of viewpoint 0, and the vertical offset of pixels indicates that the CTB3 of viewpoint 2 is shifted d2 rows down based on the CTB3 of viewpoint 0. Then, based on the horizontal offset of pixels in the reference information for viewpoint 2, each pixel in the decoded data of the CTB3 of viewpoint 0 is shifted d1 columns to the left. After the shift, the pixels in the left d1 column are deleted, and the blank pixels in the right d1 column can be filled by pixels extracted from the decoded data of the CTB3 of viewpoint 1. Similarly, based on the vertical offset of pixels in the reference information for viewpoint 2, each pixel in the decoded data of the CTB3 of viewpoint 0 is shifted d2 rows down. After the shift, the pixels in the lower d2 row are deleted, and the blank pixels in the upper d2 row can be filled by pixels extracted from the decoded data of the CTB0 of viewpoint 0.
[0127] For example, assuming CTB3 is the image block to be decoded, and the encoded data of CTB3 includes reference information for viewpoint 2, and the viewpoint identification information included in the reference information for viewpoint 2 is the identifier of viewpoint 0, then it can be determined that the target independent encoded viewpoint referenced by viewpoint 2 is viewpoint 0. Therefore, the decoded data of CTB3 for viewpoint 0 is obtained by decoding the encoded information of CTB3 for viewpoint 0. Assuming that the reference information for viewpoint 2 also includes the rotation angle offset of the pixels, the rotation angle offset of the pixels indicates that the CTB3 of viewpoint 2 is rotated clockwise by θ degrees based on the CTB3 of viewpoint 0. At this point, in the CTB3 decoding data of viewpoint 0, for a square pixel block with four vertex coordinates (30,40), (30,80), (60,40), and (60,80), after rotating the square pixel block clockwise by θ degrees, the positions of its four vertices change. Assuming the top vertex coordinates are (33,35), the bottom vertex coordinates are (57,85), the leftmost vertex coordinates are (28,75), and the rightmost vertex coordinates are (62,45), then based on the four vertex coordinates of the rotated square pixel block, the vertex coordinates of the square area where the pixel points need to be acquired are determined to be (28,35), (28,85), (62,35), and (62,85). The pixel points within this square area are acquired, and based on the pixel values of the acquired pixel points, the decoding data corresponding to the square pixel block with vertex coordinates (30,40), (30,80), (60,40), and (60,80) in CTB3 of viewpoint 2 is calculated. Following the method described above, the decoding data of CTB3 at viewpoint 0 can be adjusted based on the rotation angle offset of the pixels to obtain the decoding data of CTB3 at viewpoint 2.
[0128] Through the above decoding process, the decoded data of N image blocks are obtained, and the decoded data of each image block includes the decoded data of the above multiple perspectives.
[0129] S703 stitches the decoded data from each viewpoint into a video image, resulting in video images from multiple viewpoints.
[0130] Still assuming N is 6, for example, for viewpoint 1, from the decoded data of multiple viewpoints, obtain CTB0, CTB1, CTB2, CTB3, CTB4, and CTB5 of viewpoint 1. Then, stitch CTB0, CTB1, CTB2, CTB3, CTB4, and CTB5 of viewpoint 1 together to form the video image of viewpoint 1. Using the same method, video images from multiple viewpoints can be obtained. After decoding multiple viewpoints of a single frame of video image, decoding multiple viewpoints of the next frame of video image can continue. Here, a single frame of video image refers to video images from multiple viewpoints acquired at the same time.
[0131] The video playback device can control the display to show video images from the user's selected perspective.
[0132] Compared to independently encoding and compressing video images from multiple perspectives, the video playback device of this application embodiment does not need to independently decode the reference information of non-independent encoded perspectives contained in the encoded data. Instead, it can obtain the decoded data of the image block of the corresponding target independent encoded perspective by referring to the decoded data of the image block of the non-independent encoded perspective. Therefore, it can reduce the amount of computation in the decoding process, save the computing resources consumed in the video decoding process, and reduce the power consumption of the video playback device.
[0133] In this embodiment, video images from multiple perspectives are encoded by the same video encoding module, and the encoded compressed video stream is transmitted to the video decoding module for decoding to obtain video images from multiple perspectives.
[0134] For example, such as Figure 8 As shown, assuming four video images from different perspectives need to be transmitted, and these four images are encoded using the same video encoding module, which can be located in the video source device, the video encoding module adopts... Figure 3 or Figure 4 The video encoding method shown encodes video images from four perspectives to obtain a compressed video stream. The compressed video stream is transmitted to a video decoding module via wired or wireless means, where it is decoded by the same video encoding module. This video decoding module can be installed in the video playback device. Figure 7 The video decoding method shown decodes the compressed video stream to obtain video images from four perspectives. These multiple perspectives are decoded almost in parallel, resulting in smoother panorama switching.
[0135] In some embodiments, the video source device and the video playback device may be located in the same device, namely the video generation module and the video playback module in the same device.
[0136] Encoding video images from multiple perspectives using the same video encoding module and decoding the compressed video stream using the same video decoding module consumes fewer equipment resources compared to encoding and decoding video images from different perspectives independently, thus saving equipment costs.
[0137] The video encoding method provided in this application can be applied not only to panoramic video but also to VR or AR. Compared with traditional independent encoding of each video stream, this application can improve the compression rate of multi-view video, reduce the bit rate, and save transmission bandwidth and storage space by reducing redundant information of image blocks at the same position between viewpoints. The same transmission channel can transmit video images from more viewpoints, allowing viewers to watch the video more smoothly when switching viewpoints. Alternatively, the same transmission channel can transmit higher resolution video images, allowing viewers to enjoy clearer video.
[0138] Furthermore, in the implementation of the codec in this application embodiment, due to the adoption of fine-grained encoding at the CTB level, some viewpoints can refer to the encoding information of the CTB at the same position in other viewpoints, so that CTBs of multiple viewpoints can share memory, CTB-level encoding information, etc., which can improve the performance of video encoding equipment and reduce the occupied (double data rate, DDR) channel bandwidth and power consumption.
[0139] Based on the same inventive concept as the above embodiments, this application also provides a video encoding device, which can be understood as... Figure 8 The video encoding module shown can be applied to Figure 1 In the video source device 100 shown, the video encoding device can be used to implement the functions of the above method embodiments, and thus can achieve the beneficial effects of the above method embodiments.
[0140] The video encoding apparatus may include an image segmentation unit and an encoding unit. The video encoding apparatus is used to implement the above. Figure 3 The function shown in the method embodiment. When the video encoding device is used to implement Figure 3 The function of the method embodiment shown is as follows: the image segmentation unit can be used to execute S301, and the encoding unit can be used to execute S302 and S303. For example: the image segmentation unit is used to acquire video images from multiple perspectives at the same time and divide each video image into N image blocks; where N is an integer greater than 1. The encoding unit is used to generate encoded data of N image blocks based on the video images from multiple perspectives, and obtain a video compressed stream from multiple perspectives based on the encoded data of the N image blocks; wherein the encoded data of the N image blocks includes encoded data of at least one target image block; the encoded data of the target image block includes encoding information of the target image block of an independent encoded perspective and reference information of a non-independent encoded perspective; the reference information of the non-independent encoded perspective is used to instruct the target image block of the non-independent encoded perspective to be decoded with reference to the encoding information of the target image block of the independent encoded perspective; both the independent encoded perspective and the non-independent encoded perspective are included in the multiple perspectives.
[0141] In one possible implementation, image blocks at the same location in each video image are numbered the same; the encoding unit can specifically be used to, according to the numbering order of N image blocks, sequentially take each numbered image block as the image block to be encoded, and generate the encoded data of the image block to be encoded in the following manner: encode the image block to be encoded in the first view among multiple views to obtain the encoding information of the first view; wherein, the first view is an independent encoding view; for each non-first view among multiple views other than the first view, if the non-first view is a non-independent encoding view, then generate reference information for the non-first view; if the non-first view is an independent encoding view, then generate the encoding information of the image block to be encoded in the non-first view.
[0142] In one possible implementation, the encoding unit may specifically be used to: compare the image block to be encoded from a non-first viewpoint with the image block to be encoded from each determined independent encoding viewpoint, and determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint; if there is an independent encoding viewpoint whose degree of difference meets a set degree of difference threshold, then determine the non-first viewpoint as a non-independent encoding viewpoint; generate reference information for the non-first viewpoint based on the information of the candidate independent encoding viewpoints, wherein the candidate independent encoding viewpoints refer to independent encoding viewpoints whose degree of difference meets a set degree of difference threshold.
[0143] In one possible implementation, the encoding unit may specifically be used to: if a candidate independent encoding view includes an independent encoding view, then use the candidate independent encoding view as the target independent encoding view; and generate reference information for a non-first view based on the view identification information of the target independent encoding view.
[0144] In one possible implementation, the coding unit can be specifically used to: if the candidate independent coding view includes multiple independent coding view, then take the independent coding view with the smallest difference as the target independent coding view; and generate reference information for non-first view based on the view identification information of the target independent coding view.
[0145] In one possible implementation, the encoding unit may specifically be used to: determine the pixel offset of the image block to be encoded from a non-first viewpoint relative to the image block to be encoded from the target independent encoding viewpoint; and generate reference information containing viewpoint identification information and pixel offset of the target independent encoding viewpoint, wherein the reference information is reference information for a non-first viewpoint.
[0146] In one possible implementation, the encoding unit may specifically be used to: compare the image blocks to be encoded from non-first viewpoints with the image blocks to be encoded from each determined independent encoding viewpoint, and determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint; if there is no independent encoding viewpoint whose degree of difference meets the set degree of difference threshold, then determine the non-first viewpoint as an independent encoding viewpoint; encode the image blocks to be encoded from the non-first viewpoints to obtain the encoding information of the image blocks to be encoded from the non-first viewpoints.
[0147] Based on the same inventive concept as the above embodiments, this application also provides a video decoding device, which can be understood as... Figure 8 The video decoding module shown can be applied to Figure 1 In the video playback device shown, the video decoding device can be used to implement the functions of the above method embodiments, and thus can achieve the beneficial effects of the above method embodiments.
[0148] The video decoding device may include a data receiving unit and a decoding unit. The video decoding device is used to implement the above... Figure 7 The function shown in the method embodiment. When the video decoding device is used to implement Figure 7 The function of the method embodiment shown is as follows: the data receiving unit can be used to execute S701, and the decoding unit can be used to execute S702 and S703. For example, the data receiving unit is used to receive video compressed streams from multiple perspectives and obtain the encoded data of N image blocks contained in the video compressed stream, where N is an integer greater than 1; the encoded data of the N image blocks is obtained by encoding video images from multiple perspectives at the same time; the encoded data of the N image blocks includes the encoded data of at least one target image block; the encoded data of the target image block includes the encoding information of the target image block of the independent encoded perspective and the reference information of the non-independent encoded perspective; the reference information of the non-independent encoded perspective is used to instruct the target image block of the non-independent encoded perspective to be decoded with reference to the encoding information of the target image block of the independent encoded perspective; both the independent encoded perspective and the non-independent encoded perspective are included in multiple perspectives. The decoding unit is used to decode the encoded data of the N image blocks to obtain the decoded data of the N image blocks, and the decoded data of each image block includes the decoded data of multiple perspectives; the decoded data of each perspective are spliced together to form a video image to obtain a video image of multiple perspectives.
[0149] In one possible implementation, the decoding unit is specifically used to: sequentially treat each numbered image block as an image block to be decoded according to the numbering order of N image blocks, and obtain the decoding data of the image block to be decoded in the following manner: decode the encoding information of the independent coded viewpoint contained in the encoding data of the image block to be decoded to obtain the decoding data of the independent coded viewpoint; if the encoding data of the image block to be decoded contains reference information of the non-independent coded viewpoint, then determine the target independent coded viewpoint referenced by the non-independent coded viewpoint based on the viewpoint identification information contained in the reference information of the non-independent coded viewpoint; obtain the decoding data of the non-independent coded viewpoint based on the decoding data of the target independent coded viewpoint; the decoding data of the target independent coded viewpoint is obtained by decoding the encoding information of the image block to be decoded of the target independent coded viewpoint.
[0150] In one possible implementation, the reference information for the non-independent coding viewpoint includes pixel offsets. Specifically, the decoding unit can be used to: adjust the decoding data of the target independent coding viewpoint based on the pixel offsets to obtain the decoding data of the non-independent coding viewpoint.
[0151] Based on the same inventive concept as the above embodiments, this application also provides a video encoding device, which can be... Figure 1 The video source device 100 shown, or, can be applied to Figure 1 The video source device 100 shown can be used to implement the functions of the above method embodiments, thus achieving the beneficial effects of the above method embodiments.
[0152] Figure 9 A schematic diagram of a video encoding device is shown. See also Figure 9 As shown, the video encoding device 900 may include a processor 901 and a transceiver 902 connected to the processor 901.
[0153] Optionally, the processor 901 and the transceiver 902 can be interconnected via a bus. The processor 901 can be a general-purpose processor, such as a microprocessor, or other conventional processor. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc.
[0154] In some embodiments, the video encoding apparatus may further include a memory or external memory. The memory is used to store program instructions and data, etc. Specifically, the program instructions may include program code, which includes computer operation instructions. The memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.
[0155] Processor 901 is configured to read program instructions from memory and execute the following steps: acquire video images from multiple perspectives at the same time, and divide each video image into N image blocks; generate encoded data for N image blocks based on the video images from multiple perspectives, and obtain a video compressed stream from multiple perspectives based on the encoded data of the N image blocks; wherein N is an integer greater than 1; the encoded data of the N image blocks includes encoded data of at least one target image block; the encoded data of the target image block includes encoded information of the target image block of the independent encoded perspective and reference information of the non-independent encoded perspective; the reference information of the non-independent encoded perspective is used to instruct the target image block of the non-independent encoded perspective to be decoded with reference to the encoded information of the target image block of the independent encoded perspective; both the independent encoded perspective and the non-independent encoded perspective are included in the multiple perspectives.
[0156] Transceiver 902 is used to send compressed video streams from multiple perspectives under the control of processor 901.
[0157] In one possible implementation, image blocks at the same location in each video image are numbered the same; the processor 901 can specifically be used to, according to the numbering order of N image blocks, sequentially take each numbered image block as the image block to be encoded, and generate the encoded data of the image block to be encoded in the following way: encode the image block to be encoded in the first view among multiple views to obtain the encoded information of the first view; wherein, the first view is an independent encoded view; for each non-first view among multiple views other than the first view, if the non-first view is a non-independent encoded view, then generate reference information for the non-first view; if the non-first view is an independent encoded view, then generate the encoded information of the image block to be encoded in the non-first view.
[0158] In one possible implementation, the processor 901 may specifically be used to: compare the image block to be encoded from a non-first viewpoint with the image block to be encoded from each determined independent encoding viewpoint, and determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint; if there is an independent encoding viewpoint whose degree of difference meets a set degree of difference threshold, then determine the non-first viewpoint as a non-independent encoding viewpoint; generate reference information for the non-first viewpoint based on the information of the candidate independent encoding viewpoints, wherein the candidate independent encoding viewpoints refer to independent encoding viewpoints whose degree of difference meets a set degree of difference threshold.
[0159] In one possible implementation, the processor 901 can be specifically used to: if a candidate independent coding view includes an independent coding view, then use the candidate independent coding view as the target independent coding view; and generate reference information for a non-first view based on the view identification information of the target independent coding view.
[0160] In one possible implementation, the processor 901 can be specifically used to: if the candidate independent coding perspective includes multiple independent coding perspectives, then take the independent coding perspective with the smallest difference as the target independent coding perspective; and generate reference information for non-first perspectives based on the perspective identification information of the target independent coding perspective.
[0161] In one possible implementation, the processor 901 may specifically be used to: determine the pixel offset of the image block to be encoded from a non-first viewpoint relative to the image block to be encoded from the target independent encoding viewpoint; and generate reference information containing viewpoint identification information and pixel offset of the target independent encoding viewpoint, wherein the reference information is reference information for a non-first viewpoint.
[0162] In one possible implementation, the processor 901 may specifically be used to: compare the image blocks to be encoded from non-first viewpoints with the image blocks to be encoded from each determined independent encoding viewpoint, and determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint; if there is no independent encoding viewpoint whose degree of difference meets the set degree of difference threshold, then determine the non-first viewpoint as an independent encoding viewpoint; encode the image blocks to be encoded from the non-first viewpoints to obtain the encoding information of the image blocks to be encoded from the non-first viewpoints.
[0163] Based on the same inventive concept as the above embodiments, this application also provides a video decoding device, which can be... Figure 1 The video playback device 200 shown, or, can be applied to Figure 1 The video playback device 200 shown can be equipped with a video decoding device to implement the functions of the above-described method embodiments, thus achieving the beneficial effects of the above-described method embodiments.
[0164] Figure 10A schematic diagram of a video decoding device is shown. See also Figure 10 As shown, the video encoding device 1000 may include a processor 1001 and a transceiver 1002 connected to the processor 1001.
[0165] Optionally, the processor 1001 and the transceiver 1002 can be interconnected via a bus. The processor 1001 can be a general-purpose processor, such as a microprocessor, or other conventional processor. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc.
[0166] In some embodiments, the video decoding apparatus may further include a memory or an external memory. The memory is used to store program instructions and data, etc. Specifically, the program instructions may include program code, which includes computer operation instructions. The memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.
[0167] Transceiver 1002 is used to receive video compressed streams from multiple perspectives under the control of processor 1001.
[0168] Processor 1001 is configured to read program instructions from memory and execute the following steps: acquire encoded data of N image blocks contained in the video compressed stream, where N is an integer greater than 1; the encoded data of the N image blocks is obtained by encoding video images from multiple perspectives at the same time; the encoded data of the N image blocks includes encoded data of at least one target image block; the encoded data of the target image block includes encoding information of the target image block with independent encoding perspective and reference information of non-independent encoding perspective; the reference information of non-independent encoding perspective is used to instruct the target image block of non-independent encoding perspective to decode with reference to the encoding information of the target image block with independent encoding perspective; both independent and non-independent encoding perspectives are contained in multiple perspectives. The encoded data of the N image blocks is decoded to obtain decoded data of N image blocks, each image block's decoded data including decoded data from multiple perspectives; the decoded data of each perspective are concatenated into a video image to obtain a video image with multiple perspectives.
[0169] In some embodiments, the video decoding device may further include a display, through which the decoded video image can be displayed.
[0170] In one possible implementation, the processor 1001 can specifically be used to: sequentially treat each numbered image block as an image block to be decoded according to the numbering order of N image blocks, and obtain the decoding data of the image block to be decoded in the following manner: decode the encoding information of the independent coded viewpoint contained in the encoding data of the image block to be decoded to obtain the decoding data of the independent coded viewpoint; if the encoding data of the image block to be decoded contains reference information of the non-independent coded viewpoint, then determine the target independent coded viewpoint referenced by the non-independent coded viewpoint based on the viewpoint identification information contained in the reference information of the non-independent coded viewpoint; obtain the decoding data of the non-independent coded viewpoint based on the decoding data of the target independent coded viewpoint; the decoding data of the target independent coded viewpoint is obtained by decoding the encoding information of the image block to be decoded of the target independent coded viewpoint.
[0171] In one possible implementation, the reference information for the non-independent coding view includes pixel offsets. Specifically, the processor 1001 can be used to: adjust the decoded data of the target independent coding view based on the pixel offsets to obtain decoded data for the non-independent coding view.
[0172] Based on the same inventive concept as the above embodiments, this application also provides a video transmission system, which may include... Figure 9 The video encoding device shown and Figure 10 The video decoding device shown. In some embodiments, the video transmission system may also be understood to include... Figure 1 The video source device 100 and video playback device 200 shown are described. The video source device 100 adopts... Figure 3 The video encoding method shown encodes video images from multiple perspectives, and the video playback device 200 uses... Figure 7 The video decoding method shown decodes encoded data from multiple perspectives.
[0173] Based on the above embodiments, this application also provides a computer program product, including a computer program or instructions, which, when the computer program or instructions are run on a computer, cause the computer to execute any of the video encoding or video decoding methods provided in the above embodiments.
[0174] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to perform any of the video encoding or video decoding methods provided in the above embodiments.
[0175] The storage medium can be any available medium that a computer can access. For example, but not limited to, a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
[0176] Based on the above embodiments, this application also provides a chip, which is used to read a computer program stored in a memory and implement any of the video encoding or video decoding methods provided in the above embodiments.
[0177] Based on the above embodiments, this application provides a chip system including a processor for supporting a computer device in implementing the functions involved in any of the nodes in the above embodiments. In one possible design, the chip system further includes a memory for storing necessary programs and data of the computer device. The chip system may be composed of chips or may include chips and other discrete devices.
[0178] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0179] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0182] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of protection of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method of video coding, the method comprising: The method includes: Acquire video images from multiple perspectives at the same time, and divide each video image into N image blocks; where N is an integer greater than 1. Based on the video images from the multiple viewpoints, encoded data for N image blocks is generated; the encoded data for the N image blocks includes encoded data for at least one target image block; the encoded data for the target image block includes encoded information for the target image block from an independent encoded viewpoint and reference information for a non-independent encoded viewpoint; the reference information for the non-independent encoded viewpoint is used to instruct the target image block from the non-independent encoded viewpoint to be decoded with reference to the encoded information of the target image block from the independent encoded viewpoint; both the independent encoded viewpoint and the non-independent encoded viewpoint are included in the multiple viewpoints; Based on the encoded data of the N image blocks, a video compressed stream from the multiple perspectives is obtained.
2. The method of claim 1, wherein, Image blocks at the same location in each video image have the same number; the generation of encoded data for N image blocks based on the video images from the multiple viewpoints includes: According to the numbering order of the N image blocks, each numbered image block is taken as the image block to be encoded, and the encoded data of the image block to be encoded is generated in the following way: The image block to be encoded in the first of the multiple viewpoints is encoded to obtain the encoding information of the first viewpoint; the first viewpoint is an independent encoding viewpoint; For each non-first viewpoint among the multiple viewpoints other than the first viewpoint, if the non-first viewpoint is a non-independent coding viewpoint, then reference information for the non-first viewpoint is generated; if the non-first viewpoint is an independent coding viewpoint, then coding information for the image block to be coded for the non-first viewpoint is generated.
3. The method of claim 2, wherein, If the non-first viewpoint is a non-independent coded viewpoint, then generating reference information for the non-first viewpoint includes: The image blocks to be encoded from the non-first viewpoint are compared with the image blocks to be encoded from each determined independent encoding viewpoint to determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint. If there is an independent coding perspective whose difference meets the set difference threshold, then the non-first perspective is determined to be a non-independent coding perspective. Reference information for the non-first view is generated based on information from candidate independent coding perspectives; the candidate independent coding perspectives refer to independent coding perspectives whose difference meets a set difference threshold.
4. The method of claim 3, wherein, The generation of reference information for the non-first perspective based on information from candidate independent coded perspectives includes: If the candidate independent coding perspective includes an independent coding perspective, then the candidate independent coding perspective is taken as the target independent coding perspective; Based on the viewpoint identification information of the target independent coded viewpoint, reference information for the non-first viewpoint is generated.
5. The method of claim 3, wherein, The generation of reference information for the non-first perspective based on information from candidate independent coded perspectives includes: If the candidate independent coding perspective includes multiple independent coding perspectives, then the independent coding perspective with the smallest difference is taken as the target independent coding perspective. Based on the viewpoint identification information of the target independent coded viewpoint, reference information for the non-first viewpoint is generated.
6. The method according to claim 4 or 5, characterized in that, The generation of reference information for the non-first viewpoint based on the viewpoint identification information of the target independent coded viewpoint includes: Determine the pixel offset of the image block to be encoded from the non-first viewpoint relative to the image block to be encoded from the target independent encoding viewpoint; Generate viewpoint identification information containing the target's independent coded viewpoint and reference information containing the pixel offset; the reference information is the reference information for the non-first viewpoint.
7. The method according to any one of claims 2 to 6, characterized in that, If the non-first viewpoint is an independent coded viewpoint, then the encoding information of the image block to be encoded for the non-first viewpoint is generated, including: The image blocks to be encoded from the non-first viewpoint are compared with the image blocks to be encoded from each determined independent encoding viewpoint to determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint. If there is no independent coding perspective whose difference meets the set difference threshold, then the non-first perspective is determined to be an independent coding perspective; The image blocks to be encoded from the non-first viewpoints are encoded to obtain the encoded information of the image blocks to be encoded from the non-first viewpoints.
8. A method of video decoding, comprising: The method includes: The system receives a video compressed stream from multiple viewpoints and acquires the encoded data of N image blocks contained in the video compressed stream; where N is an integer greater than 1; the encoded data of the N image blocks is obtained by encoding video images from multiple viewpoints at the same time; the encoded data of the N image blocks includes the encoded data of at least one target image block; the encoded data of the target image block includes the encoding information of the target image block of an independent encoded viewpoint and reference information of a non-independent encoded viewpoint; the reference information of the non-independent encoded viewpoint is used to instruct the target image block of the non-independent encoded viewpoint to be decoded with reference to the encoding information of the target image block of the independent encoded viewpoint; both the independent encoded viewpoint and the non-independent encoded viewpoint are included in the multiple viewpoints; The encoded data of the N image blocks are decoded to obtain the decoded data of the N image blocks; the decoded data of each image block includes the decoded data of the multiple viewpoints; The decoded data from each viewpoint are stitched together to form a video image, thus obtaining the video images from the multiple viewpoints.
9. The method of claim 8, wherein, Decoding the encoded data of the N image blocks to obtain the decoded data of the N image blocks includes: According to the numbering order of the N image blocks, each numbered image block is taken as the image block to be decoded, and the decoding data of the image block to be decoded is obtained in the following way: The encoding information of the independent encoding viewpoint contained in the encoding data of the image block to be decoded is decoded to obtain the decoded data of the independent encoding viewpoint; If the encoded data of the image block to be decoded contains reference information of a non-independent encoded viewpoint, then the target independent encoded viewpoint referenced by the non-independent encoded viewpoint is determined based on the viewpoint identification information contained in the reference information of the non-independent encoded viewpoint. Based on the decoding data of the target independent coding perspective, the decoding data of the non-independent coding perspective is obtained; the decoding data of the target independent coding perspective is obtained by decoding the coding information of the image block to be decoded in the target independent coding perspective.
10. The method of claim 9, wherein, The reference information of the non-independent coding perspective includes pixel offsets; the process of obtaining the decoded data of the non-independent coding perspective based on the decoded data of the target independent coding perspective includes: Based on the pixel offset, the decoding data of the target independent encoding view is adjusted to obtain the decoding data of the non-independent encoding view.
11. A video encoding device, characterized in that, Includes processor and transceiver; The processor is used to: acquire video images from multiple perspectives at the same time, and divide each video image into N image blocks; where N is an integer greater than 1; Based on the video images from the multiple viewpoints, encoded data for N image blocks is generated; the encoded data for the N image blocks includes encoded data for at least one target image block; the encoded data for the target image block includes encoded information for the target image block from an independent encoded viewpoint and reference information for a non-independent encoded viewpoint; the reference information for the non-independent encoded viewpoint is used to instruct the target image block from the non-independent encoded viewpoint to be decoded with reference to the encoded information of the target image block from the independent encoded viewpoint; both the independent encoded viewpoint and the non-independent encoded viewpoint are included in the multiple viewpoints; Based on the encoded data of the N image blocks, a video compressed stream from the multiple perspectives is obtained; The transceiver is used to: transmit the compressed video streams from the multiple perspectives under the control of the processor.
12. The apparatus according to claim 11, characterized in that, Image blocks at the same location in each video image have the same number; the processor is specifically used for: According to the numbering order of the N image blocks, each numbered image block is taken as the image block to be encoded, and the encoded data of the image block to be encoded is generated in the following way: The image block to be encoded in the first of the multiple viewpoints is encoded to obtain the encoding information of the first viewpoint; the first viewpoint is an independent encoding viewpoint; For each non-first viewpoint among the multiple viewpoints other than the first viewpoint, if the non-first viewpoint is a non-independent coding viewpoint, then reference information for the non-first viewpoint is generated; if the non-first viewpoint is an independent coding viewpoint, then coding information for the image block to be coded for the non-first viewpoint is generated.
13. The apparatus according to claim 12, characterized in that, The processor is specifically used for: The image blocks to be encoded from the non-first viewpoint are compared with the image blocks to be encoded from each determined independent encoding viewpoint to determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint. If there is an independent coding perspective whose difference meets the set difference threshold, then the non-first perspective is determined to be a non-independent coding perspective. Reference information for the non-first view is generated based on information from candidate independent coding perspectives; the candidate independent coding perspectives refer to independent coding perspectives whose difference meets a set difference threshold.
14. The apparatus according to claim 13, characterized in that, The processor is specifically used for: If the candidate independent coding perspective includes an independent coding perspective, then the candidate independent coding perspective is taken as the target independent coding perspective; Based on the viewpoint identification information of the target independent coded viewpoint, reference information for the non-first viewpoint is generated.
15. The apparatus according to claim 13, characterized in that, The processor is specifically used for: If the candidate independent coding perspective includes multiple independent coding perspectives, then the independent coding perspective with the smallest difference is taken as the target independent coding perspective. Based on the viewpoint identification information of the target independent coded viewpoint, reference information for the non-first viewpoint is generated.
16. The apparatus according to claim 14 or 15, characterized in that, The processor is specifically used for: Determine the pixel offset of the image block to be encoded from the non-first viewpoint relative to the image block to be encoded from the target independent encoding viewpoint; Generate viewpoint identification information containing the target's independent coded viewpoint and reference information containing the pixel offset; the reference information is the reference information for the non-first viewpoint.
17. The apparatus according to any one of claims 12 to 16, characterized in that, The processor is specifically used for: The image blocks to be encoded from the non-first viewpoint are compared with the image blocks to be encoded from each determined independent encoding viewpoint to determine the degree of difference between each independent encoding viewpoint and the non-first viewpoint. If there is no independent coding perspective whose difference meets the set difference threshold, then the non-first perspective is determined to be an independent coding perspective; The image blocks to be encoded from the non-first viewpoints are encoded to obtain the encoded information of the image blocks to be encoded from the non-first viewpoints.
18. A video decoding device, characterized in that, Includes processor and transceiver; The transceiver is used to: receive video compressed streams from multiple perspectives under the control of the processor; The processor is configured to: acquire encoded data of N image blocks contained in a video compressed stream; wherein N is an integer greater than 1; the encoded data of the N image blocks is obtained by encoding video images from multiple perspectives at the same time; the encoded data of the N image blocks includes encoded data of at least one target image block; the encoded data of the target image block includes encoded information of the target image block from an independent encoded perspective and reference information of a non-independent encoded perspective; the reference information of the non-independent encoded perspective is used to instruct the target image block of the non-independent encoded perspective to be decoded with reference to the encoded information of the target image block from the independent encoded perspective; both the independent encoded perspective and the non-independent encoded perspective are included in the multiple perspectives; The encoded data of the N image blocks are decoded to obtain the decoded data of the N image blocks; the decoded data of each image block includes the decoded data of the multiple viewpoints; The decoded data from each viewpoint are stitched together to form a video image, thus obtaining the video images from the multiple viewpoints.
19. The apparatus according to claim 18, characterized in that, The processor is specifically used for: According to the numbering order of the N image blocks, each numbered image block is taken as the image block to be decoded, and the decoding data of the image block to be decoded is obtained in the following way: The encoding information of the independent encoding viewpoint contained in the encoding data of the image block to be decoded is decoded to obtain the decoded data of the independent encoding viewpoint; If the encoded data of the image block to be decoded contains reference information of a non-independent encoded viewpoint, then the target independent encoded viewpoint referenced by the non-independent encoded viewpoint is determined based on the viewpoint identification information contained in the reference information of the non-independent encoded viewpoint. Based on the decoding data of the target independent coding perspective, the decoding data of the non-independent coding perspective is obtained; the decoding data of the target independent coding perspective is obtained by decoding the coding information of the image block to be decoded in the target independent coding perspective.
20. The apparatus according to claim 19, characterized in that, The reference information for the non-independent coded viewpoint includes pixel offsets; the processor is specifically used for: Based on the pixel offset, the decoding data of the target independent encoding view is adjusted to obtain the decoding data of the non-independent encoding view.
21. A video transmission system, characterized in that, include: The video encoding apparatus as described in any one of claims 11 to 17; as well as The video decoding apparatus as described in any one of claims 17 to 20.
22. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 7, or to perform the method as described in any one of claims 8 to 10.
23. A computer program product, characterized in that, It includes computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 7, or to perform the method as described in any one of claims 8 to 10.
Citation Information
Patent Citations
Coding concept allowing efficient multi-view / layer coding
CN105308968A
