Multi-view video coding and decoding method and device, and storage medium

Abstract

The invention provides a multi-view video coding and decoding method and device and a storage medium, relates to the technical field of video processing, solves the problem that a multi-view video coding algorithm is high in complexity and not beneficial to real-time picture output in the prior art, and provides a better multi-view video coding and decoding method. Multiple viewpoints are simplified into parallax parameter vectors of a reference viewpoint and other viewpoints through a neural network model, single-viewpoint coding of the reference viewpoint and transmission of the parallax parameter vectors are carried out, and multi-viewpoint content is reconstructed through the neural network model at a decoding end, so that the size of a coding code stream can be reduced, bandwidth flow is saved, and the decoding efficiency is improved. And real-time picture output is facilitated.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a multi-view video encoding and decoding method, device and storage medium. Background Art

[0002] Multi-view video is a video that captures multiple perspectives of the same object or scene in a real scene, or is generated by using a virtual camera to capture a virtual 3D scene model. Multi-view video captures multiple angles of an object or scene, which contains rich three-dimensional information and can provide users with a sense of stereo and interactivity. To this end, multi-view video requires strong interactivity, so the real-time performance of the picture is very important.

[0003] Since the acquisition process is complicated, most of the content required by users needs to be downloaded to the user terminal through network transmission. Therefore, how to transmit quickly and maintain quality is very important. In the existing multi-view video encoding technology, all viewpoints are input into the encoder, and then the similarity between viewpoints is used to predict and transform to remove parallax redundancy. The above encoding process needs to process all viewpoint videos, and the amount of data required to be transmitted in the encoding transmission is large. The encoding algorithm is relatively complex, which is not conducive to real-time output of the picture. Summary of the invention

[0004] The present application provides a multi-view video encoding and decoding method, device and storage medium, which solves the problem in the related art that the complexity of the multi-view video encoding algorithm is high and is not conducive to real-time output of the picture. The present application can reduce the order of magnitude of encoding, thereby reducing the size of the encoded code stream and saving bandwidth traffic.

[0005] In a first aspect, the present application provides a multi-view video encoding and decoding method, which is applied to a transmitter in video transmission, wherein the transmitter is provided with a first neural network model, and the first neural network model is used to generate a disparity parameter vector, and the disparity parameter vector is used to represent visual difference information between different viewpoints on each image frame. The multi-view video encoding and decoding method includes:

[0006] Acquire multi-view video data, and determine view video sequences corresponding to different viewpoints from the multi-view video data;

[0007] Selecting a reference viewpoint from multiple viewpoints, and extracting a viewpoint video sequence corresponding to the reference viewpoint;

[0008] Based on parameter information associated with the encoding mode and the first neural network model, determine a target disparity parameter vector between a viewpoint video sequence corresponding to other viewpoints and a viewpoint video sequence corresponding to the associated reference viewpoint, wherein the target disparity parameter vector is used to represent visual difference information of the other viewpoints relative to the reference viewpoint on each image frame;

[0009] The viewpoint video sequence corresponding to the reference viewpoint and the target disparity parameter vector are added to the target video data, and the target video data is sent to the receiving end.

[0010] In a second aspect, the present application further provides a multi-view video encoding and decoding method, which is applied to a receiving end in video transmission, and the receiving end is provided with a second neural network model, and the second neural network model is used to restore a viewpoint video sequence corresponding to another viewpoint according to a disparity parameter vector and a viewpoint video sequence. The multi-view video encoding and decoding method includes:

[0011] receiving and decoding target video data to determine a viewpoint video sequence and a corresponding target disparity parameter vector in the received target video data;

[0012] Determining a reference viewpoint based on the acquired viewpoint video sequence;

[0013] Based on the parameter information associated with the encoding method, the target disparity parameter vector and the viewpoint video sequence corresponding to the reference viewpoint, restoring the viewpoint video sequence corresponding to other viewpoints through the second neural network model;

[0014] The multi-viewpoint video data is restored according to the viewpoint video sequence corresponding to the reference viewpoint and the viewpoint video sequences corresponding to other viewpoints.

[0015] In a third aspect, the present application further provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When one or more programs are executed by one or more processors, the one or more processors implement the multi-view video encoding and decoding method of the present application.

[0019] In a fourth aspect, the present application also provides a storage medium storing computer executable instructions, which, when executed by a processor, are used to execute the multi-view video encoding and decoding method of the present application.

[0020] The present application scheme provides a more optimal multi-viewpoint video encoding and decoding method, which simplifies multiple viewpoints into disparity parameter vectors of reference viewpoints and other viewpoints through a neural network model, and then encodes the reference viewpoint and transmits it together with the disparity parameter vector. At the decoding end, the multi-viewpoint content is reconstructed through a neural network model, which can reduce the size of the encoded bitstream, save bandwidth traffic, and facilitate real-time output of the picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of a multi-view video transmission process in the related art;

[0022] Figure 2 A schematic diagram of a reference structure of a multi-view video encoding method in the prior art;

[0023] Figure 3 A schematic diagram of the steps of a multi-view video encoding and decoding method provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the steps of a multi-view encoding and decoding method provided by another embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of a neural network model based on generative adversarial network training provided in one embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only the parts related to the embodiments of the present application rather than all structures are shown in the accompanying drawings, and those skilled in the art should be able to think of it after reading the specification of this application that as long as the technical features do not contradict each other, any combination of the technical features can constitute an optional implementation method.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable when appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated before and after are in an "or" relationship. In the description of the present application, "multiple" means two or more, and "several" means one or more.

[0029] Multi-view video captures multiple angles of an object scene, which contains rich three-dimensional information, can provide users with a sense of stereo and interactivity, and can be applied in various scenarios such as 3D TV playback, free viewpoint TV playback, virtual reality interaction, etc. The multi-view video transmission process is as follows: Figure 1 As shown, Figure 1The figure is a schematic diagram of the multi-view video transmission process in the related art, taking the virtual reality 3D scene model as the shooting scene as an example, using a virtual camera to collect videos from different viewpoints to generate multi-view videos. It can be imagined that for real scenes, binocular cameras and other shooting devices can be used to collect and obtain corresponding multi-view videos.

[0030] The multi-view video is pre-processed and encoded to generate a compressed video stream, which is then transmitted to the user terminal through the network. The user terminal uses a video decoder to decode the video stream into a multi-view video, which is then rendered and displayed on the user display terminal as required.

[0031] The encoding and decoding transmission of multi-viewpoint videos is an important link for end users to experience stereoscopic interactivity. Most of the content required by users needs to be downloaded to the user terminal through network transmission, so how to quickly transmit it while maintaining quality is very important. In addition, these application scenarios generally require strong interactivity, so the real-time performance of the picture is very important.

[0032] Video encoding and decoding mainly utilizes the spatial redundancy and temporal redundancy in video images, and uses prediction, transformation and other technologies to remove related redundancy to achieve the effect of compressing the amount of data. In the multi-view video transmission process, the size of the encoded video stream directly affects the speed of network transmission, which in turn affects the frame rate and quality of the rendered image seen by the end user, as well as the real-time performance of the image that can provide an important sense of immersion. Multi-view videos have several times more data than general single-view videos, so how to compress multi-view videos more efficiently has become a topic that needs to be studied urgently.

[0033] Figure 2 Schematic diagram of the reference structure of the multi-view video encoding method in the prior art, such as Figure 2 As shown in the figure, the reference structure of the inter-frame prediction is complex. When the existing technology transmits multiple viewpoint videos, all viewpoints are input into the encoder, and then the redundant information between viewpoints is removed through the reference structure between viewpoints. It can be understood that this scheme predicts the difference between image frames of different viewpoints (such as residual information). Although it can remove parallax redundancy, its order of magnitude is still at the image level. The encoder still needs to process multiple video information, and the amount of encoded data is still large, which affects the encoding efficiency.

[0034] In this regard, the present application provides a multi-view video encoding and decoding method, which can be applied to electronic devices. The electronic devices can implement the multi-view video encoding and decoding method of the present application to encode and decode multi-view videos, thereby transmitting and receiving multi-view videos.

[0035] It is conceivable that the electronic device includes a processor and a memory. The processor and the memory may be connected via a bus or other means. In this regard, the multi-view video encoding and decoding method in the embodiment of the present application applied by the electronic device is executed by a processor, the number of which is one or more, and the processor may include a central processing unit (CPU), a microcontroller unit (MCU) and / or a field programmable gate array (FPGA) and other processing units.

[0036] The memory, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the multi-view video encoding and decoding method in the embodiment of the present application. The memory can store a computer program required for at least one function, data created according to the use of the electronic device, etc. At present, at least a computer program related to the multi-view video encoding and decoding method is stored in the memory, so that the processor can implement the multi-view video encoding and decoding method based on the computer program stored in the memory.

[0037] Figure 3 A schematic diagram of the steps of a multi-view video encoding and decoding method provided in an embodiment of the present application, the method is applied to the sending end in video transmission, and the sending end is provided with a first neural network model. For an electronic device, when the electronic device needs to send a multi-view video to another device, the electronic device is the sending end in video transmission. In this regard, a first neural network model is provided in the electronic device, wherein the first neural network model is used to generate a disparity parameter vector, and the disparity parameter vector is used to represent the visual difference information between different viewpoints in each image frame, such as the difference in shooting angle between other viewpoints and a reference viewpoint, the difference in shooting time between other viewpoints and a reference viewpoint, etc. Based on this, the multi-view video encoding and decoding method of the present application can utilize a neural network model to reduce viewpoints and the amount of information, and optimize the amount of encoded data and real-time performance. The specific steps include steps S110-S140:

[0038] Step S110: Acquire multi-viewpoint video data, and determine viewpoint video sequences corresponding to different viewpoints from the multi-viewpoint video data.

[0039] It is conceivable that the acquired multi-view video can be multi-view video data acquired by the electronic device through the software program thereon in the constructed virtual 3D scene, and it can also be multi-view video data corresponding to the real scene acquired by the electronic device through the configured shooting device thereon or through an external shooting device. The method for acquiring and displaying rendered multi-view video can adopt the existing technology, which will not be described in detail. After the electronic device acquires the multi-view video data, it can be understood that the image frame sequence corresponding to the same viewpoint is used as the viewpoint video sequence of the viewpoint, and the multi-view video data includes viewpoint video sequences corresponding to multiple viewpoints. Accordingly, the electronic device divides the multi-view video according to the viewpoint to determine the viewpoint video sequences corresponding to different viewpoints.

[0040] Step S120 : selecting a reference viewpoint from multiple viewpoints, and extracting a viewpoint video sequence corresponding to the reference viewpoint.

[0041] It is understandable that the number of reference viewpoints may be one or more, which is determined according to the selection strategy of the reference viewpoints. Optionally, in one embodiment, when the number of viewpoints corresponding to the multi-viewpoint video data is 2 or more, correspondingly, one viewpoint may be arbitrarily selected from all viewpoints as a reference viewpoint, and the other viewpoints are associated with the viewpoint and all use the viewpoint as a reference viewpoint. Exemplarily, taking the case where the number of viewpoints corresponding to the multi-viewpoint video data is 2, such as including viewpoint I and viewpoint II, if viewpoint I is used as a reference viewpoint, viewpoint II is associated with viewpoint I and uses viewpoint I as a reference viewpoint.

[0042] In addition, in some embodiments, an arrangement order is set between viewpoints, such as sorting multiple viewpoints according to the shooting angle of each viewpoint, the virtual space position corresponding to the viewpoint when the virtual camera collects the viewpoint, etc., to determine the corresponding arrangement order. It is conceivable that the arrangement order can also be arranged at random to form a corresponding sequence. In this regard, in the arrangement order, each viewpoint is searched according to the arrangement order, and then the current viewpoint can be used as the reference viewpoint associated with the next viewpoint, wherein the current viewpoint is the viewpoint currently searched, and the next viewpoint is the viewpoint that is located after the current viewpoint in the arrangement order. Exemplarily, taking the number of viewpoints corresponding to the multi-viewpoint video data as 3 as an example, such as including viewpoint I, viewpoint II and viewpoint III, the arrangement order is viewpoint I, viewpoint II, viewpoint III, in this regard, viewpoint I is the reference viewpoint associated with viewpoint II, and viewpoint II is the reference viewpoint associated with viewpoint III.

[0043] In addition, in some embodiments, an arrangement order is set between viewpoints. In the arrangement order, each viewpoint is searched according to the arrangement order, and then all viewpoints located before the current viewpoint in the arrangement order can be used as reference viewpoints associated with the current viewpoint. For example, taking the number of viewpoints corresponding to the multi-viewpoint video data as 3 as an example, such as including viewpoint I, viewpoint II and viewpoint III, the arrangement order is viewpoint I, viewpoint II, viewpoint III respectively. In this regard, viewpoint I and viewpoint II are used as reference viewpoints associated with viewpoint III.

[0044] Step S130, based on parameter information associated with the encoding method and the first neural network model, determine the target disparity parameter vector between the viewpoint video sequence corresponding to other viewpoints and the viewpoint video sequence corresponding to the associated reference viewpoint, the target disparity parameter vector being used to represent the visual difference information of other viewpoints relative to the reference viewpoint on each image frame.

[0045] It can be understood that the encoding methods include lossless encoding and lossy encoding. When performing lossless encoding, the sender performs lossless compression without destroying any video information, and the receiver can directly restore the video image; when performing lossy encoding, the sender needs to provide corresponding parameters as parameter information, such as using compression parameters as parameter information, so that the receiver can consider the loss content of the encoding process, so as to be able to restore the video image more effectively when reconstructing the video image.

[0046] In this regard, in one embodiment, if the encoding method adopted is lossless encoding, accordingly, the electronic device can determine that the encoding method adopted is lossless encoding according to the provided parameter information, and then the electronic device uses the first neural network model thereon to take the selected reference viewpoint as a reference to determine the corresponding target disparity parameter vector for the viewpoint video sequence corresponding to other viewpoints, wherein the target disparity parameter vector is used to represent the visual difference information of other viewpoints relative to the reference viewpoint on each image frame.

[0047] After training, the first neural network model can output corresponding disparity parameter vectors according to the visual difference information between different viewpoints in each image frame. Based on this, the viewpoint video sequence corresponding to the reference viewpoint and the viewpoint video sequence corresponding to other viewpoints are used as input parameters of the first neural network model, and the first neural network model can determine the target disparity parameter vector between the viewpoint video sequence corresponding to other viewpoints and the viewpoint video sequence corresponding to the associated reference viewpoint. In this regard, the electronic device can represent the difference between other viewpoints and the reference viewpoint through the target disparity parameter vector, for example, the difference between other viewpoints and the reference viewpoint in shooting angle, the difference between other viewpoints and the reference viewpoint in shooting time, etc., so as to reduce the number while enabling the receiving end to restore the image.

[0048] Optionally, in one embodiment, if the encoding method adopted is lossy encoding, correspondingly, the electronic device can determine that the encoding method adopted is lossy encoding according to the provided parameter information, and the compression parameters corresponding to the lossy encoding are also recorded in the parameter information. In this regard, when it is determined that the encoding method is lossy encoding, the electronic device needs to obtain the compression parameters corresponding to the lossy encoding from the parameter information. Furthermore, the viewpoint video sequence and compression parameters corresponding to the reference viewpoint are used as input parameters of the first neural network model. After training, the first neural network model can output the corresponding disparity parameter vector according to the visual difference information between different viewpoints on each image frame, that is, the viewpoint video sequence corresponding to the reference viewpoint is processed by the first neural network model according to the compression parameters, and the visual difference information of other viewpoints relative to the compressed reference viewpoint on each image frame is determined to generate a disparity parameter vector. It can be understood that when performing lossy encoding, the viewpoint video sequence corresponding to the reference viewpoint needs to generate a compressed video stream through lossy encoding. Correspondingly, after the compression parameters are clarified, the viewpoint video sequence corresponding to the reference viewpoint after lossy encoding is determined in combination with the compression parameters, so that the first neural network model can output the corresponding disparity parameter vector.

[0049] Step S140: Add the viewpoint video sequence corresponding to the reference viewpoint and the target disparity parameter vector to the target video data, and send the target video data to the receiving end.

[0050] Then, the reference viewpoint is represented by the viewpoint video sequence corresponding to the reference viewpoint, and the other viewpoints are represented by the target disparity parameter vector. The two are added to the target video data. The electronic device can encode the target video data and send it to the receiving end. It can be imagined that video encoding and decoding can directly use the existing 2D video encoding and decoding method, which will not be elaborated in detail.

[0051] It can be imagined that when transmitting multiple viewpoint videos in the related technology, all viewpoints are input into the encoder, and then the redundant information between the viewpoints is removed through the reference structure between the viewpoints, such as predicting the residual information of each pixel block between image frames between different viewpoints. Although the parallax redundancy can be removed, its order of magnitude is still at the image level. The encoder still needs to process multiple video information, and the amount of encoded data is still large, which affects the encoding efficiency.

[0052] Although the encoding and decoding of the reference viewpoint in the present application is still at the image level, for other viewpoints, a disparity parameter vector can be used. The order of magnitude is smaller than that of the image level. Compared with the residual information of each pixel block, the amount of encoded data for encoding the disparity parameter vector is smaller, which makes the encoded bitstream smaller, can effectively save bandwidth traffic, and can also improve encoding efficiency, which is beneficial to the real-time output of the picture at the receiving end.

[0053] In some embodiments, the disparity parameter vector may be a set of vectors or a separate vector. In this regard, when transmitting, the actual size of the disparity parameter vector may be used to determine how to transmit, such as inserting into a video stream or transmitting separately. Optionally, in one embodiment, the first video encoding data is data after encoding the viewpoint video sequence corresponding to the reference viewpoint. Based on this, the target disparity parameter vector may be inserted into the first video encoding data, for example, the target disparity parameter vector may be inserted before or after the data segment corresponding to the first video encoding data, and the target video data may be sent to the receiving end in the form of a video stream. Therefore, by inserting the target disparity parameter vector, the receiving end can receive the target disparity parameter vector together when receiving the video stream, so that the viewpoint video sequence corresponding to other viewpoints can be quickly restored after decoding. In addition, optionally, in one embodiment, all target disparity parameter vectors may be added to the vector data, and then the first video encoding data and the vector data may be sent to the receiving end in the form of different data packets. In this regard, the receiving end can determine the corresponding viewpoint video sequence and the target disparity parameter vector by parsing the data packet, so that the viewpoint video sequence corresponding to other viewpoints can be restored.

[0054] Figure 4 A schematic diagram of the steps of a multi-view encoding and decoding method provided for another embodiment of the present application, the method can be applied to a receiving end in video transmission, the receiving end is provided with a second neural network model, for an electronic device, when the electronic device receives a multi-view video sent by another device, the electronic device is the receiving end in the video transmission. In this regard, a second neural network model is provided in the electronic device, wherein the second neural network model is used to restore the viewpoint video sequence corresponding to another viewpoint based on the disparity parameter vector and the viewpoint video sequence. Based on this, the multi-viewpoint video encoding and decoding method of the present application can use the neural network model to restore the viewpoint video sequence in a timely manner, which helps to ensure the real-time performance of the picture, and the specific steps include steps S210-step S240:

[0055] Step S210 : receiving and decoding target video data to determine a viewpoint video sequence and a corresponding target disparity parameter vector in the received target video data.

[0056] After receiving the target video data sent by the transmitter, the electronic device as the receiving end decodes the data and determines the viewpoint video sequence and the target disparity parameter vector from it. It can be imagined that the decoding performed by the receiving end is the opposite process of the encoding performed by the transmitting end. The receiving end restores the viewpoint video sequence and the target disparity parameter vector sent by the transmitting end by decoding the received data. The target disparity parameter vector is the visual difference information between different viewpoints in each image frame determined by the transmitting end.

[0057] Step S220: Determine a reference viewpoint based on the acquired viewpoint video sequence.

[0058] When the acquired viewpoint video sequence is clear, the electronic device can determine the viewpoint corresponding to the viewpoint video sequence as the reference viewpoint. It is conceivable that corresponding identification information can be set for different viewpoints so that the electronic device can determine the viewpoint according to the identification information. After the electronic device determines the reference viewpoint, the viewpoint video sequence of other viewpoints corresponding to the target disparity parameter vector can be restored through the second neural network model.

[0059] Step S230: Based on the parameter information associated with the encoding method, the target disparity parameter vector and the viewpoint video sequence corresponding to the reference viewpoint, the viewpoint video sequence corresponding to other viewpoints is restored through the second neural network model.

[0060] It can be understood that the encoding methods include lossless encoding and lossy encoding. When performing lossless encoding, the sender performs lossless compression without destroying any video information, and the receiver can directly restore the video image; when performing lossy encoding, the sender needs to provide corresponding parameters as parameter information, such as using compression parameters as parameter information, so that the receiver can consider the loss content of the encoding process, so as to be able to restore the video image more effectively when reconstructing the video image.

[0061] In this regard, in one embodiment, if the encoding method used is lossless encoding, accordingly, the electronic device can determine that the encoding method used is lossless encoding according to the provided parameter information, and then the electronic device uses the second neural network model thereon to take the selected reference viewpoint as a reference, and then combines the corresponding target disparity parameter vector. It can be imagined that the target disparity parameter vector is used to represent the visual difference information of other viewpoints relative to the reference viewpoint on each image frame. In this regard, after the second neural network model is trained, it can restore the viewpoint video sequence corresponding to other viewpoints based on the reference viewpoint according to the visual difference information between different viewpoints on each image frame.

[0062] Optionally, in one embodiment, if the adopted encoding method is lossy encoding, correspondingly, the electronic device can determine that the adopted encoding method is lossy encoding according to the provided parameter information, and the compression parameters corresponding to the lossy encoding are also recorded in the parameter information. In this regard, when it is determined that the encoding method is lossy encoding, the electronic device needs to obtain the compression parameters corresponding to the lossy encoding from the parameter information in order to restore the viewpoint video sequence corresponding to the reference viewpoint. Then, the viewpoint video sequence corresponding to the reference viewpoint, the target disparity parameter and the compression parameter are used as input parameters of the second neural network model, and the second neural network model is used to perform image restoration on the viewpoint video sequence corresponding to the restored reference viewpoint by utilizing the visual difference information of other viewpoints relative to the compressed reference viewpoint on each image frame, so as to restore the viewpoint video sequence corresponding to other viewpoints.

[0063] Step S240: restore the multi-viewpoint video data according to the viewpoint video sequence corresponding to the reference viewpoint and the viewpoint video sequences corresponding to other viewpoints.

[0064] After restoring the viewpoint video sequences corresponding to other viewpoints, the electronic device can restore the multi-viewpoint video by combining the viewpoint video sequence corresponding to the reference viewpoint and the viewpoint video sequences corresponding to other viewpoints for the user to watch. The method for acquiring and displaying the rendered multi-viewpoint video can adopt the existing technology, which will not be described in detail.

[0065] It can be seen from the above scheme that, during the transmission process, the amount of encoded data for encoding the disparity parameter vector is smaller than the residual information of each pixel block, which makes the encoded bitstream smaller and can effectively save bandwidth traffic. For the receiving end, the decoding bitstream is small and the decoding speed is faster. In addition, the receiving end can use the neural network model to restore other viewpoints based on the disparity parameter vector, reference viewpoint and parameter information, and can quickly complete the decoding and restoration of the image, which is beneficial to the real-time output of the receiving end.

[0066] In some embodiments, the first neural network model and the second neural network model are trained in a joint training manner, using the constructed first neural network model to generate a disparity parameter vector for the data in the training set, and then reconstructing the corresponding image through the second neural network model, so as to correct the first neural network model and the second neural network model according to the difference between the original image and the reconstructed image by comparing the original image and the reconstructed image. Specifically, the joint training process is as follows: Step S310-Step S330:

[0067] Step S310: input the encoded image frame corresponding to the reference viewpoint and the disparity parameter vector output by the first neural network model into the second neural network model to generate a reconstructed image frame through the second neural network model.

[0068] Step S320 : comparing the original image frame and the reconstructed image frame of the viewpoint corresponding to the disparity parameter vector to determine image difference data.

[0069] Step S330: correcting the first neural network model and the second neural network model based on the image difference data.

[0070] It can be understood that the training set includes image frames corresponding to the reference viewpoint, image frames corresponding to other viewpoints, and parameter information. When training, the first neural network model uses the data in the training set as input parameters to output the disparity parameter vector through the first neural network model, and the output of the first neural network model is used as the input parameter of the second neural network model. In addition, another input parameter of the second neural network model is the encoded image frame corresponding to the reference viewpoint, and then the reconstructed image frame is generated by the second neural network model. It can be imagined that the reconstructed image frame is a reconstruction of the image frame corresponding to other viewpoints. In this regard, during the training process, each time a reconstructed image frame is generated, it is necessary to compare the original image frame and the reconstructed image frame of the viewpoint corresponding to the disparity parameter vector (i.e., the other viewpoints mentioned above) to determine the image difference data, so as to use the image difference data to correct the first neural network model and the second neural network model.

[0071] In this regard, the first neural network model after training is deployed at the transmitting end, and the second neural network model after training is deployed at the receiving end. In some embodiments, the electronic device is a device for sending and receiving multi-viewpoint videos, which can simultaneously deploy the first neural network model and the second neural network model.

[0072] In one embodiment, the first neural network model and the second neural network model are jointly trained by using a Generative Adversarial Network (GAN), such as Figure 5 As shown, Figure 5A schematic diagram of the structure of a neural network model based on a generative adversarial network training provided in one embodiment of the present application, wherein the generative adversarial network includes a generative subnetwork and a discriminative subnetwork, wherein the generative subnetwork is used to reconstruct image frames, and the generative subnetwork includes an encoder, a first neural network model G1, and a second neural network model G2. The discriminative subnetwork is used to compare the reconstructed image frame with the original image frame, and the discriminative subnetwork includes a discriminator. It can be understood that the discriminator is a virtual module, which is used to implement the corresponding function, that is, output the discrimination result according to the input original image and the image provided by the generative subnetwork. Optionally, in one embodiment, the discriminator is a virtual module, which includes functions such as a convolution layer, batch normalization, leaky ReLU activation function, a fully connected layer, and a sigmoid activation function, so as to form a virtual module in the processor.

[0073] It is understandable that the number of viewpoints used in the training process is not limited. For ease of understanding, the following example is explained using the number of viewpoints n = 2 as an example. The constructed first neural network model and the second neural network model are trained by using the viewpoint video sequences corresponding to the reference viewpoint T1 and other viewpoints T2 and the parameter information training set.

[0074] like Figure 5 As shown, by inputting the reference viewpoint T1, other viewpoints T2 and parameter information QP of the training set into the first neural network model G1, a disparity parameter vector V of other viewpoints T2 relative to the reference viewpoint T1 is generated. And the viewpoint video sequence corresponding to the reference viewpoint T1 is encoded by the corresponding encoder, and the encoded reference viewpoint T1 is represented by viewpoint T1′.

[0075] Then, the disparity parameter vector V, the decoded viewpoint T1' and the parameter information QP are input into the second neural network G2 to generate the reconstructed viewpoint T2'. The image frames in the viewpoint video sequence corresponding to the reconstructed viewpoint T2' and the original image frames in the viewpoint video sequence corresponding to the other viewpoint T2 are input into the discriminant network to determine whether the two match according to the result of the discriminator, and then update the parameters of the corresponding neural network model, alternately iterate, and finally complete the training of the first neural network model and the second neural network model.

[0076] Specifically, for the two types of coding methods, lossy coding and lossless coding, the difference between the two methods in the training process is the reconstructed viewpoint T1′. For lossless coding, the reconstructed viewpoint T1′ is lossless and the same as the original reference viewpoint T1; while for lossy coding, the reconstructed viewpoint T1′ is lossy. Therefore, in the training process, it is necessary to train with the viewpoint T1′ obtained after lossy coding. In the training process of the generative adversarial network, through the adversarial training of the two sub-networks, the training process is simple and easy to control, which can effectively improve the training efficiency of the neural network model. In addition, by using the adversarial training of the generative sub-network and the discriminative sub-network, the natural connection of the generated data is effectively established, making the data generated by the trained neural network model more natural and realistic.

[0077] It should be noted that the method for training the neural network model can be to use other deep learning models such as generative adversarial networks or stereo radiance fields (SRF).

[0078] In addition, the image difference data can also be determined by a loss function. Optionally, in one embodiment, the reconstructed image frame generated by the second neural network model is discriminated by a loss function, that is, the difference between the original image frame in the training set and the acquired reconstructed image frame is calculated by a loss function, for example, the image difference between the original image frame and the reconstructed image frame is minimized by using loss functions such as mean squared loss (Mean Squared Loss) and mean absolute error loss (Mean Absolute Error Loss), thereby determining the corresponding image difference data, so as to correct the first neural network model and the second neural network model, thereby continuously completing alternating iterations and finally completing joint training. It should be noted that, in one embodiment, the above-mentioned discriminant subnetwork can be replaced by a loss function to also play the role of discriminating the two image frames, thereby updating the parameters of the corresponding neural network model, alternating iterations, and finally completing the training of the first neural network model and the second neural network model.

[0079] Figure 6The structural diagram of an electronic device provided in an embodiment of the present application is used to execute the multi-view coding and decoding method provided in the above embodiment, and has the functional modules and beneficial effects corresponding to the execution method. As shown in the figure, the electronic device also includes a processor 301, a memory 302, an input device 303 and an output device 304. The number of processors 301 can be one or more, and one processor 301 is taken as an example in the figure; the processor 301, the memory 302, the input device 303 and the output device 304 can be connected by a bus or other means, and the figure takes the connection through a bus as an example. The memory 302, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the multi-view coding and decoding method in the embodiment of the present application. The processor 301 executes the corresponding various functional applications and data processing by running the software programs, instructions and modules stored in the memory 302, that is, the above-mentioned multi-view coding and decoding method is realized.

[0080] The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data recorded or created during use, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 302 may further include a memory remotely arranged relative to the processor 301, and these remotely arranged memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0081] The input device 303 can be used to input corresponding digital or character information to the processor 301, and generate key signal input related to the user settings and function control of the device; the output device 304 can be used to send or display key signal output related to the user settings and function control of the device.

[0082] An embodiment of the present application further provides a storage medium storing computer executable instructions, which, when executed by a processor, are used to perform relevant operations in the multi-view video encoding and decoding method provided in any embodiment of the present application.

[0083] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0084] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0085] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A multi-view video encoding and decoding method, characterized in that: The method is applied to a transmitting end in video transmission, wherein the transmitting end is provided with a first neural network model, wherein the first neural network model is used to generate a disparity parameter vector, wherein the disparity parameter vector is used to represent visual difference information between different viewpoints on each image frame, and the multi-view video encoding and decoding method comprises: Acquire multi-view video data, and determine view video sequences corresponding to different viewpoints from the multi-view video data; Selecting a reference viewpoint from a plurality of viewpoints, and extracting a viewpoint video sequence corresponding to the reference viewpoint; Based on parameter information associated with the encoding mode and the first neural network model, determine a target disparity parameter vector between a viewpoint video sequence corresponding to other viewpoints and a viewpoint video sequence corresponding to the associated reference viewpoint, wherein the target disparity parameter vector is used to represent visual difference information of other viewpoints relative to the reference viewpoint on each image frame; The viewpoint video sequence corresponding to the reference viewpoint and the target disparity parameter vector are added to target video data, and the target video data is sent to a receiving end.

2. The multi-view video encoding and decoding method according to claim 1, characterized in that: The selecting a reference viewpoint from a plurality of viewpoints and extracting a viewpoint video sequence corresponding to the reference viewpoint comprises: A viewpoint is randomly selected from multiple viewpoints as a reference viewpoint associated with other viewpoints; or, Determine an arrangement order of the corresponding multiple viewpoints, and use the current viewpoint as a reference viewpoint associated with a next viewpoint, wherein the next viewpoint is a viewpoint located one position after the current viewpoint in the arrangement order; or, An arrangement order of the corresponding multiple viewpoints is determined, and all viewpoints located before the current viewpoint in the arrangement order are used as reference viewpoints associated with the current viewpoint.

3. The multi-view video encoding and decoding method according to claim 1, characterized in that: The method of determining, based on the parameter information associated with the encoding mode and the first neural network model, a target disparity parameter vector between a viewpoint video sequence corresponding to other viewpoints and a viewpoint video sequence corresponding to the associated reference viewpoint, wherein the target disparity parameter vector is used to represent visual difference information of other viewpoints relative to the reference viewpoint on each image frame, includes: When the encoding mode is lossy encoding, obtaining compression parameters corresponding to the lossy encoding from the parameter information; The viewpoint video sequence corresponding to the reference viewpoint and the compression parameters are used as input parameters of the first neural network model, so that the viewpoint video sequence corresponding to the reference viewpoint is processed according to the compression parameters by the first neural network model, and the visual difference information of other viewpoints relative to the compressed reference viewpoint on each image frame is determined to generate the disparity parameter vector.

4. The multi-view video encoding and decoding method according to claim 1, characterized in that: The first video encoding data is data obtained by encoding the viewpoint video sequence corresponding to the reference viewpoint, and the viewpoint video sequence corresponding to the reference viewpoint and the target disparity parameter vector are added to the target video data, and the video data is sent to the receiving end, including: Inserting the target disparity parameter vector into the first video encoding data, and sending the target video data to the receiving end in a video stream manner; or, All target disparity parameter vectors are added to the vector data, and the first video encoding data and the vector data are sent to the receiving end in sequence.

5. The multi-view video encoding and decoding method according to any one of claims 1 to 4, characterized in that: The first neural network model and the second neural network model are trained by joint training, and the second neural network model is used to restore the viewpoint video sequence corresponding to another viewpoint according to the disparity parameter vector and the viewpoint video sequence; The joint training process includes: Inputting the encoded image frame corresponding to the reference viewpoint and the disparity parameter vector output by the first neural network model into the second neural network model, so as to generate a reconstructed image frame through the second neural network model; Comparing the original image frame of the viewpoint corresponding to the disparity parameter vector and the reconstructed image frame to determine image difference data; Based on the image difference data, the first neural network model and the second neural network model are modified.

6. The multi-view video encoding and decoding method according to claim 5, characterized in that: The comparing the original image frame of the viewpoint corresponding to the disparity parameter vector and the reconstructed image frame to determine image difference data includes: Based on a preset discriminator, a generative adversarial network is constructed to use the original image frame and the reconstructed image frame as input parameters of the discriminator, and the image difference data is determined by the discriminator, wherein the discriminator is used to compare the image frames in the received input parameters and determine the image difference data between the image frames; or, Based on a preset loss function, an image difference between the original image frame and the reconstructed image frame is minimized to determine the image difference data.

7. A multi-view video encoding and decoding method, characterized in that: Applied to a receiving end in video transmission, the receiving end is provided with a second neural network model, the second neural network model is used to restore a viewpoint video sequence corresponding to another viewpoint according to a disparity parameter vector and a viewpoint video sequence, and the multi-viewpoint video encoding and decoding method includes: receiving and decoding target video data to determine a viewpoint video sequence and a corresponding target disparity parameter vector in the received target video data; Determining a reference viewpoint based on the acquired viewpoint video sequence; Based on the parameter information associated with the encoding mode, the target disparity parameter vector and the viewpoint video sequence corresponding to the reference viewpoint, restoring the viewpoint video sequence corresponding to other viewpoints through the second neural network model; The multi-viewpoint video data is restored according to the viewpoint video sequence corresponding to the reference viewpoint and the viewpoint video sequences corresponding to other viewpoints.

8. The multi-view video encoding and decoding method according to claim 7, characterized in that: The method of restoring viewpoint video sequences corresponding to other viewpoints based on the parameter information associated with the encoding mode, the disparity parameter vector and the second neural network model comprises: Based on the parameter information, determining whether the encoding mode is lossy encoding; When the encoding mode is lossy encoding, obtaining compression parameters corresponding to the lossy encoding from the parameter information; The viewpoint video sequence corresponding to the reference viewpoint, the disparity parameter vector and the compression parameter are used as input parameters of the second neural network model, so that the viewpoint video sequence corresponding to the reference viewpoint is restored according to the disparity parameter vector and the compression parameter through the second neural network model, so as to restore the viewpoint video sequences corresponding to other viewpoints.

9. The multi-view video encoding and decoding method according to any one of claims 7 to 8, characterized in that: The first neural network model and the second neural network model are trained by joint training, the first neural network model is used to generate a disparity parameter vector, and the disparity parameter vector is used to represent visual difference information between different viewpoints on each image frame; The joint training process includes: Inputting the encoded image frame corresponding to the reference viewpoint and the disparity parameter vector output by the first neural network model into the second neural network model, so as to generate a reconstructed image frame through the second neural network model; Comparing the original image frame of the viewpoint corresponding to the disparity parameter vector and the reconstructed image frame to determine image difference data; Based on the image difference data, the first neural network model and the second neural network model are modified.

10. The multi-view video encoding and decoding method according to claim 9, characterized in that: The comparing the image frame corresponding to the reference viewpoint and the reconstructed image frame to determine the image difference data includes: Based on a preset discriminator, a generative adversarial network is constructed to use the original image frame and the reconstructed image frame as input parameters of the discriminator, and the image difference data is determined by the discriminator, wherein the discriminator is used to compare the image frames in the received input parameters and determine the image difference data between the image frames; or, Based on a preset loss function, an image difference between the original image frame and the reconstructed image frame is minimized to determine the image difference data.

11. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-view video encoding and decoding method as described in any one of claims 1-10.

12. A storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a processor, they are used to perform the multi-view video encoding and decoding method according to any one of claims 1 to 10.

Images