Image synthesis
Through neural network processing of multiple images and three-dimensional spatial data, the problem of limited viewing space in immersive videos is solved, efficient and high-quality view synthesis is achieved, and user experience and image synthesis performance is improved.
Patent Information
- Application Number
- CN202380027900.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-16
- Filing Date
- 2023-03-07
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art When providing immersive video, the 6-degree of freedom experience in the viewing space is limited. As the viewer moves, the quality of the view composite image decreases, resulting in errors and artifacts. The existing methods are complex and computationally intensive, and the data requirements are High, difficult to meet high-quality image synthesis.
A neural network is used for view synthesis. By receiving multiple images and three-dimensional spatial data, the view synthesis neural network is trained to generate view shifted images for different viewing postures, reducing data requirements, and improving image synthesis quality and user experience.
It realizes efficient and high-quality view synthesis under different viewing postures, reduces complexity and data requirements, and improves the user experience and image synthesis performance of immersive videos.
Smart Images

Figure CN118947126A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to neural network based image synthesis, and in particular, but not exclusively, to neural network based video frame synthesis. Background Art
[0002] The variety and range of image and video applications has increased significantly in recent years, with the continuous development and introduction of new services and ways to utilize and consume video.
[0003] For example, an increasingly popular service is to provide image sequences in a way that the viewer can actively and dynamically interact with the system to change the parameters of the rendering. In many applications, a very attractive feature is the ability to change the viewer's effective viewing position and viewing direction, for example allowing the viewer to move around and look around in the rendered scene.
[0004] Such a feature can in particular allow to provide a virtual reality experience to the user. This can allow the user, for example, to move (relatively) freely in a virtual environment and to dynamically change his position and where he is looking. Typically, such extended reality (XR) applications are based on a three-dimensional model of a scene, wherein the model is being dynamically evaluated to provide a specifically requested view. This approach is well known from, for example, gaming applications for computers and consoles (e.g. in the category of first-person shooters). Extended reality (XR) applications include virtual reality (VR), augmented reality (AR) and mixed reality (MR) applications.
[0005] An example of a video service or application that has been proposed is immersive video, in which video is played back on, for example, a VR head mounted device to provide a three-dimensional experience. For immersive video, the viewer has the freedom to look around and move around in the presented scene so that this can be perceived as viewing from different viewpoints. However, in many typical approaches, the amount of movement is limited to, for example, a relatively small area around a nominal viewpoint, which may typically correspond to a viewpoint from which video capture of the scene has been performed. In such applications, three-dimensional scene information is often provided that allows high quality view image synthesis for viewpoints that are relatively close to (one or more) reference viewpoints, but if the viewpoint deviates too much from the reference viewpoint, the three-dimensional scene information is degraded.
[0006] Immersive video may also often be referred to as 6 degrees of freedom (6DoF) or 3DoF+ video.MPEG Immersive Video (MIV) is an emerging standard where immersive video is enabled and standardized using metadata on top of existing video codecs.
[0007] The problem with immersive video is that the viewing space (the viewing space is the 3D space in which the viewer has a 6DoF experience of sufficient quality) is limited. As the viewer moves outside the viewing space, degradation and errors caused by the synthesized view images become increasingly significant and may result in an unacceptable user experience. Since the provided 3D video data does not provide enough information (e.g., de-occlusion data) for view synthesis, errors, artifacts and inaccuracies may occur in the generated view images in certain situations.
[0008] For example, immersive video data may be provided in the form of multiple views that may be accompanied by a depth data (MVD) representation of the scene. A scene may be captured by multiple spatially distinct cameras, and the captured images and depth maps may be provided together. However, the likelihood that such a representation does not include sufficient image data for de-occluded areas increases significantly as the viewpoint becomes increasingly different from the reference viewpoint from which the MVD data was captured. Therefore, as the viewer moves away from the nominal position, portions of the image that should be de-occluded for the new viewpoint but are missing in the source view cannot be synthesized directly from the image data describing such portions of the image. Moreover, imperfect depth maps may cause distortion when performing view synthesis, and in specific cases portions of the view distortion that are an integral part of the synthesis operation. The farther the synthesized viewpoint is from the original camera viewpoint, the more severe the distortion in the synthesized view.
[0009] Most existing methods for view synthesis from multi-view images require the availability of a depth map for each source view and synthesize a new view by combining predictions from multiple reference views. Depth-manipulated rendering is then used to perform predictions from the reference views.
[0010] A recent alternative is to convert the multi-view images into a layered representation containing transparency (e.g., multi-plane image (MPI), multi-sphere image (MSI), multi-object surface image (MOSI) format) and encode / compress the layered representation. After decoding, new views can be synthesized via back-to-front layer composition.
[0011] Regardless of the specific 3D image format used, the process of generating view images for different viewing postures is a challenging process that often results in imperfect images. Most image synthesis algorithms tend to introduce some artifacts or errors, for example, due to imperfect de-occlusion, etc. In addition, most methods tend to require complex and computationally intensive processing to generate images for different viewing postures with the desired quality. In addition, the requirements for the 3D image data that needs to be generated in order to allow the view synthesis algorithm to generate reliable and high-quality images are challenging to meet and often increase the data rate and make the distribution and communication of such data difficult and resource-demanding.
[0012] Therefore, improved methods would be advantageous. In particular, methods allowing improved operation, increased flexibility, improved immersive user experience, reduced complexity, convenient implementation, improved synthesized image quality, improved rendering, increased (possibly virtual) user movement freedom, improved and / or facilitated view synthesis for different viewing postures, reduced data requirements, and / or improved performance and / or operation would be advantageous. Summary of the invention
[0013] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.
[0014] According to one aspect of the present invention, there is provided an apparatus comprising: a first receiver arranged to receive multiple images of a three-dimensional scene captured from different viewing postures; a second receiver arranged to receive three-dimensional spatial data for the scene; a view synthesis neural network arranged to generate view-shifted images of the scene for different viewing postures based on the multiple images and the three-dimensional spatial data; a neural network trainer arranged to train the view synthesis neural network based on the images of the scene for different viewing postures; a generator arranged to generate an audio-visual data stream comprising: image data of at least some of the multiple images; scene data representing the three-dimensional spatial data; coefficient data describing coefficients of the view synthesis neural network after training.
[0015] The present invention may allow improved data characterizing the scene being generated and may allow improved image synthesis. The method may allow improved image synthesis for different viewing postures. In many embodiments, data may be generated that allows efficient and high quality view shifting and synthesis based on neural networks.
[0016] In many embodiments and scenarios, the present invention may provide an improved user experience. The method may, for example, allow improved XR / AR / VR / MR applications based on limited capture of a scene.
[0017] In many embodiments, using a neural network for view synthesis may provide improved view synthesis, and may, for example, allow for increased flexibility and reduced requirements for data representing the scene.
[0018] The multiple images may include a set of multi-view images. The images in the multiple images may be two-dimensional images. The scene may be a real-life scene, and the multiple images may be captured images of the real-life scene. The images may be images captured by an image or video camera. The three-dimensional spatial data may represent the spatial properties of the scene, and may, for example, include or consist of a 3D point cloud, a mesh or depth map of a (static) background or (dynamic) foreground object, a 3D point cloud, a (static) background or a (dynamic) foreground object, or a depth map. The three-dimensional spatial data may be independent of any viewing posture or multiple images of the multiple images / any viewing posture or multiple images of the multiple images. The three-dimensional spatial data may be viewing posture-independent. The three-dimensional spatial data may be viewing posture-independent data describing the geometric / spatial properties of the scene. The three-dimensional spatial data may not be bound to a view, or may be represented in multiple images and / or viewing postures of multiple images, or may be associated with multiple images and / or viewing postures of multiple images. The three-dimensional spatial data may include or consist of geometric primitives, such as 3D points, planes, surfaces, lines, grids, etc.
[0019] The server neural network may specifically be a convolutional network. The server neural network may include one or more convolutional layers. The coefficients may include coefficients of filters / kernels for one or more convolutional layers. One, more or all convolutional layers of the server neural network may be hidden layers or processing layers.
[0020] In some embodiments, the server neural network may include one or more fully connected layers. The coefficients may include weights for connections of the one or more fully connected layers.
[0021] A gesture may be / represent / indicate a position, an orientation, or both a position and an orientation.
[0022] The images of the scene for different viewing poses used to train the view synthesis neural network may include one or more of the plurality of images received by the first receiver.
[0023] The view-shifted image may be an image synthesized for a viewing posture different from that of the plurality of images.
[0024] According to an optional feature of the invention, the first receiver is arranged to receive a set of input video sequences, the set of input video sequences representing views of the three-dimensional scene from the different viewing postures, and the multiple images are frames of the set of input video sequences; and wherein the neural network trainer is arranged to dynamically train the view synthesis neural network so that the coefficients of the view synthesis neural network vary over time; and the generator is arranged to generate the audio-visual data stream to include frames from the set of input video sequences and time-varying coefficient data describing the time-varying coefficients of the view synthesis neural network after training.
[0025] The invention may allow improved data to dynamically characterize a scene being generated and may allow improved video synthesis.
[0026] According to an optional feature of the invention, the neural network trainer is arranged to train the view synthesis neural network using a set of training images, the set of training images comprising a first set of reference images for viewing postures not represented by images in said at least some of the plurality of images.
[0027] In many cases this may provide improved training and hence improved image synthesis.It may enhance the training to reflect properties over a larger area, such as specifically over the entire desired viewing area.
[0028] According to an optional feature of the invention, there is provided that the first set of reference images comprises at least one image selected from the group consisting of: reference images generated by non-neural network view shifting using the multiple images; reference images generated based on a visual scene model of the scene.
[0029] This can provide improved training and image synthesis over larger areas and can compensate / mitigate limited capture data available. The approach can reduce capture requirements.
[0030] According to an optional feature of the invention, the set of training images comprises a second set of reference images, the second set of reference images comprising images from the plurality of images, and the neural network trainer is arranged to apply different weights to the first set of reference images and the second set of reference images.
[0031] In many cases, this can provide improved operation and training.For example, different weights can be implemented by applying different weights for contributions to the cost function used to train the server neural network.
[0032] According to an optional feature of the invention, the neural network trainer is arranged to encode and decode at least some of the plurality of images before at least some of the plurality of images are provided to the view synthesis neural network.
[0033] In many embodiments, this may improve performance and may, for example, result in neural network view synthesis also compensating / mitigating effects and degradations caused by the communication / distribution of audiovisual data streams.
[0034] According to an optional feature of the invention, the neural network trainer is arranged to encode and decode the three dimensional spatial data before the three dimensional spatial data is provided to the view synthesis neural network.
[0035] In many embodiments, this may improve performance and may, for example, result in neural network view synthesis also compensating / mitigating effects and degradations caused by the communication / distribution of audiovisual data streams.
[0036] According to an optional feature of the invention, the neural network trainer is arranged to initialize the view synthesis neural network using a set of default coefficients and to train the view synthesis neural network to determine modified coefficients for the view synthesis neural network; and wherein the generator is arranged to include at least some of the modified coefficients in the audio-visual data stream.
[0037] In many embodiments and situations this may allow improved operation, and in many embodiments may specifically allow reduced data rates and / or faster initialization.
[0038] In some embodiments, the neural network trainer may be arranged to select the set of default coefficients from a plurality of sets of default coefficients in response to the plurality of images.
[0039] According to an optional feature of the invention, the generator is arranged to select a subset of coefficients to transmit in response to differences between the modified coefficients and the default coefficients.
[0040] In many embodiments and situations, this may allow for improved operation.
[0041] In some embodiments, the generator is arranged to include image data only for a subset of the plurality of images, and also includes at least one feature map of a view synthesis neural network for images at a viewing pose different from the viewing pose of the subset of the plurality of images.
[0042] In some embodiments the generator is arranged to include at least one feature map of a view synthesis neural network for images at a viewing pose different from the viewing pose of the plurality of images.
[0043] In some embodiments the generator is arranged to generate the audiovisual data stream such that at least two of the image data, the scene data and the coefficient data share a Group of Pictures (GOP) structure.
[0044] In some embodiments, the first receiver is arranged to receive an input video sequence, the input video sequence comprising a plurality of frames representing views of the three-dimensional scene from the different viewing postures for each of a plurality of moments in time, and the plurality of images comprising a group of frames for one moment in time; and wherein the neural network trainer is arranged to dynamically train the view synthesis neural network to cause coefficients of the view synthesis neural network to vary over time; and the generator is arranged to generate an output video sequence, the output video sequence comprising image data of at least some frames of the input video sequence and coefficient data at different moments in time of the output video sequence.
[0045] According to one aspect of the present invention, there is provided an apparatus comprising: a receiver arranged to receive an audio-visual data stream, the audio-visual data stream comprising: image data representing a plurality of images of a three-dimensional scene captured from different viewing postures, three-dimensional spatial data of the scene, and coefficient data describing coefficients of a view synthesis neural network; the view synthesis neural network arranged to generate view-shifted images of the scene for different viewing postures based on the plurality of images and the three-dimensional spatial data; and a neural network controller for setting coefficients of the neural network in response to the coefficient data.
[0046] The invention may allow improved image synthesis.The method may allow improved image synthesis for different viewing postures.
[0047] In many embodiments and scenarios, the present invention may provide an improved user experience. The method may, for example, allow improved XR / AR / VR / MR applications based on limited capture of a scene.
[0048] The plurality of images may comprise a set of multi-view images. The images in the plurality of images may be two-dimensional images. The scene may be a real-life scene, and the plurality of images may be captured images of the real-life scene. The images may be images captured by an image or video camera. The three-dimensional spatial data may represent the spatial properties of the scene, and may, for example, comprise or consist of a 3D point cloud, a mesh or depth map of a (static) background or (dynamic) foreground object.
[0049] The client neural network may specifically be a convolutional network. The client neural network may include one or more convolutional layers. The coefficients may include coefficients of filters / kernels for one or more convolutional layers. One, more or all convolutional layers of the client neural network may be hidden layers or processing layers.
[0050] In some embodiments, the client neural network may include one or more fully connected layers. The coefficients may include weights for connections of the one or more fully connected layers.
[0051] The view-shifted image may be an image synthesized for a viewing posture different from that of the plurality of images.
[0052] According to an optional feature of the invention, the audio-visual data stream comprises a set of video sequences, the set of video sequences comprising a plurality of frames representing views of the three-dimensional scene from the different viewing postures, and the plurality of images are frames of the plurality of frames; and the coefficient data comprises time-varying coefficient data describing time-varying coefficients of the view synthesis neural network; and the neural network controller is arranged to change the coefficients of the view synthesis neural network in response to the time-varying coefficient data.
[0053] In many embodiments, this may allow for improved image synthesis.
[0054] According to an optional feature of the invention, the neural network controller is arranged to: determine interpolation coefficient values for at least one moment in time, for which coefficient data is not included in the audio-visual data stream, the interpolation coefficient values being determined based on coefficient values of the time-varying coefficient data; and set the coefficients of the view synthesis neural network to the interpolation coefficient values for the at least one moment in time.
[0055] In many embodiments, this may allow for improved image synthesis.
[0056] According to an optional feature of the invention, the audio-visual data stream comprises at least one neural network feature map for images having a viewing posture different from the viewing posture of the plurality of images, and the neural network controller is arranged to configure the view synthesis neural network using the at least one feature map.
[0057] In many embodiments, this may allow improved image synthesis. In many cases, it may allow a reduced data rate for the audiovisual data stream.
[0058] In many embodiments, this may allow for improved image synthesis.
[0059] According to an optional feature of the invention, the neural network controller is arranged to initialise the view synthesis neural network with a set of default coefficients and to overwrite the default coefficients by coefficients from the audiovisual data stream.
[0060] In many embodiments, this may allow for improved image synthesis.
[0061] According to an optional feature of the invention, the neural network controller is arranged to select the set of default coefficients from a plurality of sets of default coefficients in response to the plurality of images.
[0062] In many embodiments, this may allow for improved image synthesis.
[0063] In some embodiments the neural network controller is arranged to initialise the neural network using default coefficients and to overwrite the default coefficients by coefficients determined from the coefficient data.
[0064] According to an optional feature of the invention, at least two of the image data, the scene data and the coefficient data share a group of pictures, GOP, structure in the audiovisual data stream.
[0065] In many cases this may allow improved operation.It may facilitate the encoding and / or decoding of audiovisual data streams.
[0066] According to an optional feature of the invention, the view synthesis neural network comprises a plurality of tunable layers and a plurality of non-tunable layers, and the audiovisual data stream comprises coefficient data only for the tunable layers.
[0067] In many embodiments, this may allow improved image synthesis. In many cases, it may allow a reduced data rate for the audiovisual data stream.
[0068] In some embodiments, the view synthesis neural network comprises a plurality of fixed layers having fixed coefficients and a plurality of variable layers having variable coefficients; and the neural network trainer is arranged to train only the plurality of tunable layers; and the generator is arranged to generate the audio-visual data stream to include only coefficient data for the variable coefficients.
[0069] According to an optional feature of the invention, the three-dimensional spatial data includes video point cloud data of the scene.
[0070] In many embodiments, this may allow for improved image synthesis.
[0071] According to one aspect of the present invention, there is provided an audio-visual data stream, comprising: image data representing multiple images of a three-dimensional scene captured from different viewing postures, three-dimensional spatial data of the scene, and coefficient data describing coefficients of a view synthesis neural network, wherein the view synthesis neural network is used to generate view-shifted images of the scene for different viewing postures based on the multiple images and the three-dimensional spatial data.
[0072] According to one aspect of the present invention, a method for generating an audio-visual data stream is provided, the method comprising: receiving multiple images of a three-dimensional scene captured from different viewing postures; receiving three-dimensional spatial data of the scene; a view synthesis neural network generating view shifted images of the scene for different viewing postures based on the multiple images and the three-dimensional spatial data; training the view synthesis neural network based on the images of the scene for different viewing postures; and generating an audio-visual data stream, the audio-visual data stream comprising: image data of at least some of the multiple images, scene data representing the three-dimensional spatial data, and coefficient data describing coefficients of the view synthesis neural network after training.
[0073] According to one aspect of the present invention, a method is provided, comprising: receiving an audio-visual data stream, the audio-visual data stream comprising: image data representing multiple images of a three-dimensional scene captured from different viewing postures, three-dimensional spatial data of the scene, and coefficient data describing coefficients of a view synthesis neural network; the view synthesis neural network generates view shifted images of the scene for different viewing postures based on the multiple images and the three-dimensional spatial data; and setting the coefficients of the view synthesis neural network in response to the coefficient data.
[0074] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:
[0076] Figure 1 An example of elements of an image distribution system is illustrated;
[0077] Figure 2 An example of an image capture situation is illustrated;
[0078] Figure 3 illustrates examples of elements of an apparatus according to some embodiments of the present invention;
[0079] Figure 4 illustrates examples of elements of an image synthesis apparatus according to some embodiments of the present invention;
[0080] Figure 5 illustrates an example of a view synthesis neural network;
[0081] Figure 6 illustrates examples of poses of training images used to train a view synthesis neural network;
[0082] Figure 7 An example of illustrating the structure of an artificial neural network;
[0083] Figure 8 illustrates an example of a node of an artificial neural network;
[0084] Fig. 9 Some elements of a possible arrangement of a processor for implementing elements of an apparatus according to some embodiments of the invention are illustrated. DETAILED DESCRIPTION
[0085] Three-dimensional video capture, distribution and presentation are becoming increasingly popular and desirable in some applications and services. A specific approach is referred to as immersive video, and typically includes providing a view of a real-world scene and often a real-time event, which allows small viewer movement, such as relatively small head movement and rotation. For example, a live video broadcast of, for example, a sporting event that allows for the generation of a view based on a local client after a small viewer head movement can provide the impression that a user is sitting in the stands watching the sporting event. The user can, for example, look around and will have a natural experience similar to that of the audience at that position in the stands. Recently, display devices with position tracking and 3D interaction support applications based on 3D capture of real-world scenes have become increasingly common. Such display devices are well suited for immersive video applications that provide an enhanced three-dimensional user experience.
[0086] The following description will focus on immersive video applications, but it should be understood that the principles and concepts described can be used in many other applications and embodiments.
[0087] In many approaches, immersive video can be provided locally to a viewer by, for example, a standalone device that does not use any remote video server or even have any access to any remote video server. However, in other applications, immersive applications can be based on data received from a remote or central server. For example, video data can be provided to a video rendering device from a remote central server and can be processed locally to generate a desired immersive video experience.
[0088] Figure 1 Such an example of an immersive video system is illustrated, wherein a video rendering client 101 contacts a remote immersive video server 103, for example via a network 105 such as the Internet. The server 103 may be arranged to support a potentially large number of client devices 101 simultaneously.
[0089] The server 103 may support an immersive video experience, for example, by sending three-dimensional video data describing a real-world scene. The data may specifically describe the visual features and geometric properties of the scene generated from real-time capture of the real world by a set of (possibly 3D) cameras.
[0090] In order to provide such services for real-world scenes, scenes are usually captured from different locations and using different camera capture poses. Therefore, the relevance and importance of multi-camera capture and, for example, 6DoF (6 degrees of freedom) processing are rapidly increasing. Applications include live concerts, live sports, and telepresence. The freedom to choose one's own viewpoint enriches these applications by increasing the sense of presence over regular videos. In addition, immersive scenarios can be envisioned, where viewers can navigate and interact with the live captured scenes. For broadcast applications, this may require real-time view synthesis at the client device. View synthesis introduces errors, and these errors depend on the implementation details of the algorithm.
[0091] Figure 1 The server 103 is arranged to generate an audiovisual data stream comprising data describing a scene, and the client 101 is arranged to receive and process the audiovisual data stream to generate an output video stream that dynamically reflects changes in user posture, thereby providing an immersive video experience, wherein the rendered view adapts to viewing / user posture / placement changes.
[0092] In the art, the terms placement and posture are used as common terms for position and / or direction / orientation. For example, the combination of the position and direction / orientation of an object, camera, head or view can be referred to as a posture or placement. Therefore, a placement or posture indication can include six values / components / degrees of freedom, wherein each value / component generally describes an individual property of the position / position or orientation / orientation of the corresponding object. Of course, in many cases, placement or posture can be considered or represented with fewer components, for example, if one or more components are considered to be fixed or unrelated (for example, if all objects are considered to be at the same height and have a horizontal orientation, four components can provide a complete representation of the posture of the object). In the following, the term posture is used to refer to a position and / or orientation that can be represented by one to six values (corresponding to the maximum possible degrees of freedom). The term posture can be replaced by the term placement. The term posture can be replaced by the term position and / or orientation. The term posture can be replaced by the term position and orientation (if the posture provides information about both position and orientation), by the term position (if the posture provides information about (possibly only) position) or by the orientation (if the posture provides information about (possibly only) orientation).
[0093] A common approach for representing scenes is called multi-view with depth (MVD) representation and capture. In this approach, a scene is represented by multiple images with associated depth data, where the image representations are typically from different viewing poses of a limited capture area. In practice, images can be captured using a camera rig that includes multiple cameras and a depth sensor.
[0094] exist Figure 2An example of such a capture system is shown in . The figure shows a scene to be captured, the scene comprising a scene object 201 in front of a background 203. A plurality of capture cameras 205 are positioned in the capture area 205. The result of the capture may be a representation of the 3D scene via a multi-view image and a depth representation (i.e., by providing images and depths for a plurality of capture poses). Thus, the multi-view image and the depth representation may provide a description of the 3D scene from the capture area. Thus, the data representing the 3D scene may provide a representation of the 3D scene from the capture area, and the visual data provides a description of the 3D scene from the capture area.
[0095] The MVD representation may be used to perform view synthesis, whereby an image of a view of a scene from a given viewing pose can be generated. The viewing pose may require shifting the image view of the MVD representation to the viewing pose so that an image of the view(s) of the scene from the viewing pose can be generated and rendered to the user. The view shifting and synthesis is based on depth data, where, for example, the parallax shift between a position in the MVD image(s) and the viewing pose image depends on the depth of the corresponding object in the scene.
[0096] The quality of the generated view images will depend on the image and depth information available for the view synthesis operation. It also depends on the amount of view shift required.
[0097] For example, a view shift typically results in de-occlusion of parts of an image that may not be visible in, for example, a master image for the view shift. These holes can be filled with data from other images if these images have captured the de-occluded elements, but it is also often possible that parts of the image that are de-occluded for the new viewpoint are also missing from the other source views. In this case, view synthesis requires estimating the data based on, for example, surrounding data. The de-occlusion process inherently tends to be a process that introduces inaccuracies, artifacts, and errors. Furthermore, this tends to increase with the amount of view shift, and in particular, the likelihood of missing data (holes) during view synthesis increases with the distance from the capture pose of the image.
[0098] Another source of possible distortion may be imperfect depth information. Typically, depth information is provided by a depth map, where depth values have been generated by imperfect depth estimation (e.g., by parallax estimation between source images) or measurement (e.g., ranging), and thus the depth values may include errors and inaccuracies. View shifting is based on depth information, and imperfect depth information leads to errors or inaccuracies in the synthesized image(s). The further the synthesized viewpoint is from the original camera viewpoint, the more severe the distortion in the synthesized target view image.
[0099] Therefore, as the viewing posture moves further and further away from the capture posture, the quality of the composite image will tend to deteriorate. If the viewing posture is far enough away from the capture posture, the image quality is reduced to an unacceptable level and a poor user experience may be experienced.
[0100] Recently, it has been proposed to perform view shifting and synthesis based on neural networks. In such applications, a neural network can be trained to synthesize view images for different viewing postures based on input images from a set of captured postures. The training can be based on a set of reference images provided for a series of viewing postures, wherein the neural network is arranged to generate corresponding images for the same viewing posture. The difference between the reference images and the generated corresponding images can be used to adapt and train the neural network. The trained neural network can then be used to dynamically generate view images for different viewing postures, for example, to support a user to virtually move around in a scene. In the following, a method for representing, processing and rendering views of a scene based on the use of a neural network will be described.
[0101] Figure 3 Elements of an exemplary apparatus for generating an audiovisual data stream representing a scene are illustrated. The audiovisual data stream includes data representing visual properties of the scene. The apparatus may specifically be a source or server that provides the audiovisual data stream to a remote device. Specifically, the apparatus may be Figure 1 The following description will focus on describing the device with reference to the server 103 .
[0102] Figure 4 The flow of audiovisual data for representing a scene and in particular from Figure 3 The apparatus may specifically generate an image of a scene from an audiovisual data stream generated by a remote device (such as from Figure 3 The device may be a terminal user device or client that receives the audio-visual data stream. Figure 1 The following description will focus on describing the device with reference to the server 103 .
[0103] The server 103 includes a first receiver 301 that receives multiple images of a three-dimensional scene, wherein the images are view images for different viewing postures in the scene. The multiple images can be images of the scene captured from different viewing postures. In some embodiments, the image can be an image of a real scene captured by cameras at different locations in the scene and / or with different orientations. In some embodiments, the image can alternatively or additionally be an image of a virtual or partially virtual scene. In this case, the image can be generated / captured, for example, by evaluating a virtual model of the scene. The following will focus on captured images of images of real scenes captured by cameras with different postures in the scene. The multiple images will be referred to as captured images or input images.
[0104] The set of input images may be provided in any suitable format.In some embodiments, the images may also include depth, and in particular, a set of MVD images may be received.
[0105] In some embodiments, the input image can be a static image of a static scene. However, in many embodiments, the image can be a frame of one or more video sequences. For example, for each capture position, an input video sequence can be received. For example, a dynamic scene, such as, for example, a sporting event, can be captured from a group of video cameras located at a given location. Each camera can provide a video sequence comprising frames. Therefore, the first receiver 301 can receive a plurality of video sequences comprising time-ordered frames, wherein the frames of different video sequences represent different capture poses of the scene. It should be understood that in some embodiments, a single video sequence comprising frames of different capture poses at each moment can be received.
[0106] The server 103 also includes a second receiver 303 arranged to receive three-dimensional spatial data of the scene. The spatial data may indicate the spatial and geometric properties of objects in the scene. The spatial data may specifically indicate the spatial properties of objects in the scene, such as, for example, position, orientation, extension. Such objects may include background objects of the scene. For example, the spatial data may describe the spatial extent of one or more objects in the scene.
[0107] The spatial data may be independent of the input image and may describe objects of the scene independent of the pose of the input image. The spatial data may be common to multiple input images and / or to multiple frames of a video sequence. The spatial data may specifically be a model of a three-dimensional scene.
[0108] The spatial data may specifically be one or more of a 3D point cloud, a mesh of (static) background or (dynamic) foreground objects, or a depth map. The spatial data may for example correspond to a depth map of a MVD representation.
[0109] The spatial data may specifically be spatial data provided by a laser system measuring an accurate point cloud. Alternatively, the spatial data may be provided via human-guided interaction with the source view images to interactively create a mesh, for example using a multi-view mesh editor. Alternatively, a computer graphics mesh model (e.g. of a stadium) may be used and fitted (rotated and translated) to the source view data and used as a static background model. Alternatively, an object detector for a given object class (player) or category (floor, wall, sky) may be run and the resulting class label or category label may be used.
[0110] In some embodiments, the spatial data can be "artificial" using what we call a multi-view mesh editor, in which case humans inject information (humans can be considered sensors). In many cases, another accurate source is a laser that generates a point cloud. Such lasers are typically much more accurate than "standard" depth sensors based on structured light or time of flight. As another example, an object detector can be used that, for example, finds and fits the geometry of a basket in a basketball scene. Spatial data can be data generated by a different sensor than the one that generated the input image. Spatial data can typically provide accurate geometric data of the scene. Spatial data can be geometric data of the scene.
[0111] The first receiver 301 and the second receiver 303 are coupled to a data stream generator 305, which is arranged to generate an audio-visual data stream representing a scene. The data stream generator 305 is arranged to include some or possibly all of the input images in the audio-visual data stream. In many embodiments, the data stream generator 305 can be arranged to encode the images / frames / video according to a suitable image or video encoding algorithm.
[0112] The data stream generator 305 is further arranged to generate and include scene data representing the 3D spatial data. In many embodiments, some or all of the 3D spatial data may be included in the audiovisual data stream. In some embodiments, the generation of the scene data may include encoding or compression of the 3D spatial data.
[0113] As an example, an entity graph comprising object category labels with multi-view blocks similar to what is known from the MPEG immersive video standard may be included.
[0114] The server 103 also comprises a view synthesis neural network, hereinafter referred to as server neural network 307. The server neural network 307 is arranged to synthesize images for different viewing postures based on the spatial data and some or all of the captured input images.
[0115] The server neural network 307 may be arranged to receive a set of images (which may specifically be frames of a video sequence) representing a scene from different viewing postures, and it may also receive 3D spatial data. Based on this data, the server neural network 307 may be arranged to generate an output image corresponding to a view image for a posture different from the input viewing posture.
[0116] In more detail, the server neural network 307 may be a convolutional neural network.
[0117] Before being input into the neural network, the 3D spatial data and / or the multi-view data may pass through functions / operations that depend on source view parameters and / or target view parameters.
[0118] In the following, reference will be made to Figure 5 Advantageous example methods are described.
[0119] A depth map generator 501 that generates a depth map of a target view
[0120] 3D spatial data are usually not associated with a given source viewpoint, but are known or specified in a common world coordinate system. They are usually very accurate. As an example, a single laser that captures a point cloud can be used. As a first step, a depth map can be generated by projecting this point cloud to a target view. Possible holes can be filled via interpolation and / or extrapolation.
[0121] Source View Convolutional Network 503
[0122] The network may take as input a full resolution 3 (color) channel source view image and stack a small number of convolutional layers (e.g. 2 layers), each with a stride (step size) greater than 1 to downscale the consecutive feature maps corresponding to each layer. For three layers, a stride of 4 may be useful. When the layers are stacked, the number of feature maps (= channels) may increase from 3 input layers to 8 two layers after the input (= image) layer. It is important to realize that these layers "live" in source view image coordinates. Note that, as is common, each convolution operation (which is linear) is followed by a non-linear (so-called activation) function. We recommend using rectified linear units (ReLU), but other approaches are of course possible.
[0123] Generator 505 for generating a flow field from a target view to each source view
[0124] Now that an (accurate) depth map of the target view has been generated, we can use this depth map to compute a so-called flow field, which maps each pixel in the source view to the corresponding pixel in each target view. If the source view parameters are accurately calibrated and the laser (or other device) provides accurate 3D data, this mapping will also be very accurate. However, occlusions of points in the source view are not detected, and so this needs to be resolved by the target view convolutional network (see below). The computation of the flow field and the retrieval of source view pixels to target views can be done at (sub-millisecond) speed on an average GPU.
[0125] Connection circuit 507 for connecting N source view networks to target view network 509
[0126] To allow end-to-end training, a function graph in which all functions are differentiable can be used. Since the flow field is defined from the target view back to each source view, it is differentiable and can therefore be inserted as a separate layer between other layers in the neural network. Frameworks such as PyTorch allow such "bent layers" to be defined. Specifically, for PyTorch, the layer is torch.nn.functional.grid_sample. See reference: https: / / pytorch.org / docs / stable / generated / torch.nn.functional.grid_sample.html. In this example, the output of a given source view layer (which is a 4D tensor (batch size, channel, height, width)) is extracted to the target view together with the flow field, where it is represented at a specified resolution. The resolution can be exactly the target view resolution or a lower resolution. In many cases, the resolution can be the same as the resolution of a specific layer from the source view. In a specific example, both the high-resolution source view input layer (=actual source view image) and the low-resolution output layer (final feature map) of the source view network.
[0127] Object View Convolutional Network 509
[0128] The target view network takes as input the now warped layers from (a subset of) the source view. The network takes as an additional input the depth map provided by the laser data. The method can continue in two separate branches:
[0129] 1. Depth-based hybrid weight computation branch: The branch that takes the depth map as input is followed by two convolutional layers using a 3×3 kernel with an upscaling step before each convolution. The output layer is specified to output N channels at the target view resolution, where N is the number of source views used.
[0130] 2. Image-based hybrid weight computation branch: This branch takes the distorted source view output layer (feature map) as input, upscales it to the target view resolution, convolves it with a 3x3 kernel, and also outputs an N-channel weight tensor, one per source view. Figure 1 indivual.
[0131] Each branch provides independent hints on how to perform occlusion-aware blending (based on depth changes or color differences). In this example, the simplest solution of just averaging these weights to get the final blending weight tensor is implemented. This tensor is normalized so that the sum over the channels is equal to 1, and then multiplied by the concatenated warp source view tensor. The result is summed over the channels to get the predicted target view image. Note that this is done for each warp source view color channel.
[0132] Note that for training, the source view can be used as the ground truth using a known leave-one-out strategy.
[0133] It should be noted that the above strategy of having a network in each source view has the following advantages: the source view network can compute features that will provide information about occlusion and texture changes after warping to the target. It can help better suppress occluded pixels in the blending operation.
[0134] More information about neural networks used for view synthesis operations can be found, for example, in “Advances in Neural Rendering” by Tewari et al. (Computing Research Repository (CoRR) volume = {abs / 2111.05849}, 2021, https: / / arxiv.org / abs / 2111.05849, arXiv, timestamp = {Tue, 16Nov 2021 12:12:31+0100}).
[0135] The server 103 further comprises a neural network trainer 309 arranged to train the server neural network 307 .
[0136] The neural network trainer 309 may use a training set that includes a large number of images / frames of the scene from different viewing poses.
[0137] The neural network trainer 309 may use any suitable training method, including implementing gradient descent based on a cost function determined from the training data / reference images.
[0138] In more detail, one of the available optimizers available in the framework, such as PyTorch or TensorFlow, can be used. Specifically, the so-called Adam optimizer can be used, such as described in https: / / pytorch.org / docs / stable / generated / torch.optim.Adam.html? highlight=adam#torch.optim. (Adam or Diederik P. Kingma and Jimmy Ba Adam: "A Method for Stochastic Optimization". 3rd International Conference for Learning Representations, San Diego, 2015).
[0139] Therefore, the neural network trainer 309 can train the server neural network 307 for the task of generating view images for different viewing postures. Specifically, based on the images / frames and the 3D spatial data included in the audio-visual data stream, the trained server neural network 307 is configured to generate view images / frames of the scene from different viewing postures. The coefficients / weights used for the connections to the nodes / artificial neurons and / or for the kernels / filters of the convolutional layers are adapted / trained to perform such view shifting with high quality.
[0140] In many embodiments (as will be described in more detail later), a node / neuron may have / be assigned a value determined as a weighted combination of input values from nodes of previous layers. The weighted combination has coefficients, where these coefficients are determined by training.
[0141] In many cases, node values may also include bias values, which may also be determined through a training process.In some embodiments, in addition to the coefficient data, the generated audiovisual data stream may also include data describing the bias values of the nodes.
[0142] In this method, the coefficients / weights of the trained network (or at least some of these coefficients) are fed to the data stream generator 305 and included in the audiovisual data stream. Thus, coefficient data describing the coefficients of the view synthesis neural network after training are included in the audiovisual data stream.
[0143] The coefficients may include coefficients of filters / kernels for one or more convolutional layers. The coefficients may include weights of connections for one or more fully connected layers.
[0144] The client 101 comprises a receiver 401 which receives the audiovisual data stream generated by the server 103 .
[0145] The client 101 further comprises a view synthesis neural network, which will hereinafter be referred to as the client neural network 403. The received images / frames and the received 3D spatial data are fed from the received audio-visual data stream to the client neural network 403. The client neural network 403 is then arranged to synthesize images for different viewing postures based on these images and spatial data.
[0146] The server neural network 307 may be arranged to receive a set of images (which may specifically be frames of a video sequence) for different viewing postures, and it may also receive 3D spatial data. Based on this data, the client neural network 403 may be arranged to generate output images corresponding to view images for postures different from the viewing posture of the received images.
[0147] In many cases, the audiovisual data stream may include a video sequence of frames representing views of a scene from different viewing postures. The client neural network 403 may then generate an output video sequence including frames representing views of the scene from viewing postures that may change dynamically. The output frames are generated by applying the client neural network 403 structure and operations to the received images / frames and spatial data.
[0148] The client neural network 403 and the server neural network 307 may specifically be implemented using the same structure, and thus may have the same layers, nodes / neurons, connections, etc. Comments provided with respect to the server neural network 307 also apply mutatis mutandis to the client neural network 403, and vice versa.
[0149] The client 103 further comprises a neural network controller 405 coupled to the receiver 401 and the client neural network 403. The neural network controller 405 is arranged to control the setting of coefficients / weights of the client neural network 403. In particular, the neural network controller 405 is arranged to process the received coefficient data to extract the neural network coefficients encoded in the audiovisual data stream. The client neural network 403 is then arranged to set the coefficients of the client neural network 403 according to the received coefficient data. The neural network controller 405 may set the coefficients of the client neural network 403 to the values provided in the coefficients.
[0150] Thus, neural network controller 405 may be arranged to set client neural network 403 to be identical to trained server neural network 307, and thus replicate trained server neural network 307 at client 103. Neural network controller 405 may then set client neural network 403 to function in the same manner as trained server neural network 307, thereby ensuring that client neural network 403 can perform high quality view shifting / image / frame synthesis.
[0151] The client 103 may include a viewing gesture generator 407 that generates a viewing gesture for which a view image / frame is generated. For example, the viewing gesture generator 407 may be arranged to receive a viewing gesture of a viewer (and specifically in a scene). The viewing gesture may represent the position and / or orientation of the viewer viewing the scene, and it may specifically provide a gesture for which a view of the scene should be generated. It should be understood that many different methods for determining and providing viewing gestures are known, and any suitable method may be used. For example, the viewing gesture generator 407 may be arranged to receive gesture data from a VR head-mounted device worn by a user, from an eye tracker, and the like.
[0152] The viewing gesture generator 407 may provide the viewing gesture to the client neural network 403, which then generates an output view image / frame for the given viewing gesture based on the received image and 3D spatial data.
[0153] The client neural network 403 is coupled to a renderer 409, which is arranged to render the generated image / frame(s) generated by the client neural network 403. The renderer may specifically generate a display signal representing the image / frame. The display signal may be fed to a display, which may display the generated view image / frame to a viewer / user.
[0154] In this method, view synthesis and shifting can therefore be based on neural network methods that generally result in high quality images and / or reduced complexity. In addition, the method can allow advantageous operations to be performed without any need to train the neural network on the client side. More specifically, training can be performed on the server side. This can generally allow improved training because more appropriate training data is generally available on the server side. Typically, the server side may have access to additional images of the scene that are not transmitted to the client side. In addition, although including a neural network and training operation for generating view shifting on the server side (wherein the resulting images of such operations are not required and are often discarded) may increase the complexity of the server side, this is generally greatly outweighed by the potential quality of operation and client-side improvements and the reduced computational requirements on the client side. This is generally a significant advantage because a single server can generally serve multiple and potentially many clients.
[0155] In this method, a server or other source device may encode images such as video frames, spatial data, and neural network coefficients in a bitstream. The neural network coefficients may be determined prior to encoding to maximize view synthesis quality given the limited pixel space and available spatial data in a video atlas. The video frame may be a (subset of) the received source view or a transform thereof.
[0156] As a specific example, multi-view data and depth data (eg, collected using a laser) may be input to a synthetic neural network (server neural network 307). The coefficients of the neural network may be encoded individually, or may be encoded as a video, for example, using a video atlas.
[0157] It should be appreciated that any suitable method for training server neural network 307 (and thus implicitly client neural network 403) may be used, depending on the specific preferences and requirements of an individual embodiment.
[0158] In many embodiments, the neural network trainer 309 is arranged to train the server neural network 307 using a set of training images including images received by the first receiver 301. In particular, input images (in particular such as frames of a video sequence) capturing a scene from different viewing postures may be received. Some or all of these may be provided to the data stream generator 305 and included in the audiovisual data stream. These images may be used as input to the neural network when performing view synthesis. In addition, the received captured images may be used as reference images for training the server neural network 307.
[0159] The neural network trainer 309 may specifically be arranged to train the server neural network 307 by minimizing a cost function that reflects the difference between the synthetic images generated for different viewing postures and the reference images for the same viewing posture. In some cases, the synthetic viewing posture may correspond to a viewing posture for which captured images are also received, and these images may be used as reference images.
[0160] In many embodiments, training and fitting of parameters is performed with the inputs to server neural network 307 already processed according to at least some of the processing that occurs as part of the communication to client neural network 403.
[0161] In particular, neural network trainer 309 may be arranged to encode and decode some or all of the input images before providing them as input images to client neural network 403. The encoding and decoding may be consistent with the encoding performed by data stream generator 305 and the complementary decoding performed at client 103. Thus, when client neural network 403 performs image synthesis, the images provided to server neural network 307 may more closely correspond to the images provided to client neural network 403. The encoding and decoding operations may include all operations performed as part of the transmit path, typically including, for example, priming, packing, or pixel rate reduction.
[0162] Similarly, the neural network trainer 309 may be arranged to encode and decode the three-dimensional spatial data of the scene before it is provided to the view synthesis neural network. The encoding and decoding may be the same operations as those applied to the spatial data as part of the communication to the client neural network 403. As a result, the spatial data provided to the server neural network 307 may more closely correspond to the spatial data provided to the client neural network 403.
[0163] The encoding and decoding operations may be dynamically adapted to reflect the specific encoding and decoding parameters currently being used to transmit data to the client 103 .
[0164] While the data provided to the client neural network 403 for synthesis may be processed to reflect the effects and degradation caused by communication operations, the reference image is not subjected to such processing, and the input image used as the reference image is used unchanged.
[0165] This approach may allow improved training of the server neural network 307, and in particular may allow implementation of the client neural network 403 based on received coefficients that may not only be optimized for view shifting, but may also mitigate / compensate for some effects of communication operations. In practice, for a given set of encoding parameters, the training of the neural network uses non-ideal data (image and spatial data) that has been subjected to video / image compression, but still optimizes the synthetic network to predict ideal data.
[0166] Typically, the set of captured images and capture positions is relatively limited, and this may potentially affect the accuracy of the training of the server neural network 307. In many embodiments, the neural network trainer 309 is arranged to train the server neural network 307 using a set of training images that includes reference images for viewing postures not represented by the input images.
[0167] For example, a set of training data may be generated to include viewing gestures distributed over a given area, which may correspond to a target view area, for example. Figure 6 As shown in the above example, input data can be received for five viewing postures. For example, the scene can be captured by five different cameras. In this case, input images can be provided for five different postures, and these images can be provided as input images to the server neural network 307 and can be transmitted to the client neural network 403.
[0168] However, to provide additional training data, and to represent an increased range of viewing postures, reference images may be generated for viewing postures other than the captured posture. These additional reference images may be generated to be distributed over the expected viewing area, such as Figure 6Thus, not only can additional images be used for training, but these additional images can be generated to cover a wider area, thereby providing better training of the neural network used to synthesize the images over the entire intended viewing area.
[0169] In some embodiments, the additional reference images may be generated by view shifting based on the captured input images, wherein the view shifting is not based on a neural network. Thus, for a given viewing posture in the viewing area, a view synthesis operation may be performed based on the input images to generate a view image for the viewing posture.
[0170] The view synthesis operation is performed using a view synthesis operation that is not based on the server neural network 307. It will be appreciated that many different view shifting and image synthesis algorithms are known to those skilled in the art, and any suitable approach may be used.
[0171] In some embodiments, some or all of the additional reference images may be generated by processing a visual scene model (such as a graphical model) of the scene. For example, for a virtual scene, a model may be provided that can evaluate different viewing postures to generate view images for the viewing postures. It should be understood that many different methods for generating and evaluating visual scene models are known to those skilled in the art, and any suitable method may be used.
[0172] In some embodiments, the training data may therefore include a set of reference images corresponding to the input captured images and a set of reference images for non-captured viewing poses. The images in the latter set of reference images will also be referred to as reference images to reflect that they can generally be generated by non-neural network based view shifting or model evaluation.
[0173] In such an embodiment, the neural network trainer 309 is arranged to apply different weights to different sets of reference images. In particular, when training and fitting the neural network, reference images corresponding to input images may be weighted more highly than simulated reference images. Specifically, weighting different reference images may be accomplished by weighting the images differently when generating a cost function for training (e.g., by a gradient descent method). For example, an error determined for an input image may contribute more to the cost function than a corresponding error determined for a simulated image. In many embodiments, improved training may be achieved by differential weighting for different types of reference images.
[0174] The described approach can mitigate the risk of a neural network being trained to essentially only learn to predict virtual viewing poses that are part of a captured configuration, since the source viewing poses are the only poses that the network can be trained on if only the input images are used as reference images. This problem can be mitigated by adding simulated reference images / sample data to the training set.
[0175] The method may specifically include a dataset-dependent cost function with a weight α between 0 and 1 that balances the viewing area extrapolation quality and overfitting to the captured source views. For example, the cost function may be determined as the sum of contributions, which for each image is given by:
[0176]
[0177] x is the source view image, y is the ground-truth target view image, and is the target view image as predicted by the neural network. The parameter α can be increased to give more weight to the actual captured data from the capture compared to the simulated data.
[0178] In some embodiments, the training of server neural network 307 may be based on evaluating the output using a second neural network trained to evaluate the output quality of the image of client neural network 403.
[0179] In some embodiments, the server neural network 307 and / or the client neural network 403 may be initialized with a set of default coefficients. For example, a set of default coefficients that have been found to be effective for many general scenarios and situations may be stored. The default coefficients may, for example, be shared between the server and the client (e.g., during a previous session), and thus the same default coefficients may be loaded into the server neural network 307 and the client neural network 403.
[0180] In such a system, the neural network trainer 309 may initialize the server neural network 307 with default coefficients and may then initialize a training sequence specifically based on the received video sequence / image / frame. Based on the training, modified coefficients may be determined and some or all of these coefficients may be included in the audiovisual data stream and sent to the client.
[0181] Similarly, the neural network controller 405 may initialize the client neural network 403 with default coefficients, which may be fixed coefficients or may be coefficients stored during a previous session, for example. When initializing a new session, the neural network controller 405 may accordingly be arranged to begin operation immediately without receiving the neural network coefficients from the server.
[0182] When modified coefficients are determined by the training operation, these may be included in the audiovisual data stream and when received by the client, the neural network controller 405 may proceed to set the coefficients of the client neural network 403 to these modified values. Thus, the client neural network 403 may start with default coefficients and then update the default coefficients when modified coefficients are received. In some embodiments, the neural network controller 405 is therefore arranged to initialize the neural network with default coefficients and overwrite the default coefficients with coefficients determined from the coefficient data.
[0183] In some embodiments, the data stream generator 305 may be arranged to include all modified coefficients in the audio-visual data stream when the training operation is complete. Thus, all coefficients determined for the client neural network 403 may be included. Similarly, when received at the client, the neural network controller 405 may be arranged to set all coefficients of the client neural network 403 to the values received from the server.
[0184] In some embodiments, only a subset of coefficients may be transmitted. Specifically, the data stream generator 305 may select a subset of coefficients for transmission in response to the difference between the modified coefficients and the default coefficients. For example, only modified coefficients that differ from the default (or previous) coefficients by more than a threshold may be sent to the client. Similarly, when receiving coefficient data that includes values for only a subset of coefficients, the neural network controller 405 may proceed to modify only those coefficients. This approach may reduce the data rate required to transmit the coefficients, and may reduce the data rate of the audiovisual data stream and / or allow for a faster update rate of the neural network coefficients.
[0185] In some embodiments, the neural network trainer 309 and / or the neural network controller 405 are arranged to select the default coefficients from a plurality of sets of default coefficients. For example, different sets of coefficients may be stored. The neural network trainer 309 and / or the neural network controller 405 may then select between these based on the received image.
[0186] For example, coefficients may have been determined during a previous training sequence for capturing scenes during bright summer daytime, during summer dusk, during a snowy winter day, during a cloudy and rainy fall day, etc. Corresponding default coefficient sets may be stored.
[0187] When a new session is initialized, the neural network trainer 309 and / or the neural network controller 405 may analyze one or more of the initial images to determine which stored situation is most likely represented. For example, the selection may be based on brightness and color parameters. If the image is very bright and contains many warm colors, then the coefficients for summer may be selected, if the colors are unsaturated and the brightness is relatively low, then the coefficients for autumn may be selected, and if the image is bright and the predominant color is white, then the coefficients for winter may be selected.
[0188] In some embodiments, the coefficient sets may differ for hardware type / settings (camera brand, exposure settings, etc.), etc.
[0189] This approach may provide improved initial operation and may facilitate and improve initial training as well as reduce the perceived impact of subsequent updates of the coefficients.
[0190] In many embodiments, the method may be used as described with video sequences, and thus may provide time-varying images. Each video sequence may, for example, be a temporal sequence of frames, and the neural network may be arranged to generate a frame at a given time instant by applying the neural network operation to the input frame at that time instant. For example, the frames closest (in time) to the time instant at which the frame is to be generated may be selected in each video sequence, and the neural network operation may be applied to these.
[0191] Thus, in some embodiments, the audiovisual data stream comprises time-varying video data. In many embodiments, the audiovisual data stream may comprise time-varying coefficient data for a neural network.
[0192] For example, the neural network trainer 309 may be arranged to iteratively perform training of the server neural network 307. For example, at regular intervals, the neural network trainer 309 may control the server to perform training of the server neural network 307, resulting in the generation of new updated coefficients. These updated coefficients may then be encoded and included in the audiovisual data stream. Thus, the audiovisual data stream may, for example, at regular intervals include coefficient data with new modified coefficients. The new modified coefficients may further be applied to the server neural network 307 and used as initial coefficients for the next training operation.
[0193] The client 103 may receive an audiovisual data stream having time-varying coefficient data describing the time-varying coefficients of the view synthesis neural network, and it may change the coefficients of the neural network in response to the time-varying coefficient data, typically by overwriting some or all of the coefficients of the client neural network 403.
[0194] In many embodiments, the neural network controller 405 may be arranged to update the coefficients when new data is received. However, in some embodiments, the neural network controller 405 may also be arranged to update the coefficients when no coefficients are received, i.e., to apply time-varying coefficients to the client neural network 403.
[0195] In many embodiments, the neural network controller 405 may be arranged to generate interpolated coefficients for moments in time when no coefficient data is provided. For example, when the audiovisual data stream includes a set of new coefficients for some discrete moments in time (e.g., at regular time intervals), the neural network controller 405 may interpolate the times between these moments, determining the coefficients by interpolating from the received coefficients. Thus, the client neural network 403 may be updated more frequently than new coefficients are received by overwriting the current coefficients with the interpolated coefficients.
[0196] As a specific example, the audiovisual data stream may include new coefficient data with regular intervals of, for example, 10 seconds. However, the neural network controller 405 may interpolate between the provided coefficient values to generate new modified coefficient values, for example, every second. The neural network controller 405 may then update the coefficients of the client neural network 403 every second.
[0197] This approach can allow for significantly reduced data rates and / or reduced computational burden, since time- and computationally demanding training operations can be performed less frequently, and less additional data is generated and transmitted. However, improved quality can still be achieved by providing better update rates for client neural network 403 operations.
[0198] It should be understood that any suitable method for determining the interpolation coefficients may be used. For example, interpolation may be performed separately and individually for each coefficient, and, for example, a simple linear interpolation may be performed between the received coefficient values for the time before the current moment and the received coefficient values for the time after the current moment. In other embodiments, more complex interpolation operations may be used.
[0199] In some embodiments, the coefficient data may provide values for all coefficients of the neural network. However, in some embodiments, the coefficient data may include data for only a subset of the coefficients.
[0200] In some embodiments, the neural network may be arranged to have a plurality of tunable layers and a plurality of non-tunable layers. The non-tunable layers may be layers whose coefficients are fixed and cannot be changed. Specifically, the coefficients of the non-tunable layers that are not modified or changed by training may, for example, be set to predetermined default values, where then when the neural network trainer 309 performs training of the server neural network 307, these values are fixed and not modified. For example, in the described embodiments, the source view network parameters may be assumed to be constant and known by the client, while only the target view network parameters are sent.
[0201] The tunable layers have varying coefficients and specifically include coefficients that can be modified or changed by a training operation. Thus, when training of the server neural network 307 is performed, only the coefficients of the tunable layers are changed / optimized.
[0202] In such an embodiment, the data stream generator 305 may be arranged to include coefficients only for tunable layers, but not for non-tunable layers.
[0203] The neural network controller 405 may be arranged at the client to implement the client neural network 403 to include one or more non-tunable layers with fixed coefficients, respectively. Additionally, the client neural network 403 is implemented with one or more tunable layers. The neural network controller 405 may be arranged to set the coefficients of the non-tunable layers to default values (or these may, for example, be fixedly implemented as part of the client neural network 403), and to set the coefficients of the tunable layers based on the coefficient data.
[0204] This approach can in particular be combined with a dynamic approach to time-varying coefficient data, wherein time-varying coefficient data are provided only for the tunable layers.
[0205] The use of tunable and non-tunable layers may be very advantageous for the described method, since the combination of adaptable and fixed layers is particularly suitable for distributed learning based view synthesis. In particular, in some cases, partial adaptation of training may allow significantly reduced data rates, since fewer coefficients are transmitted, but high quality results may be achieved.
[0206] In some embodiments, the server may be arranged to send a feature map of an image. For example, instead of including a particular image in the audiovisual data stream, the server may proceed to extract a set of feature values for a given layer of a convolutional neural network forming the server neural network 307.
[0207] For example, in the specific example of a convolutional network described previously, the server can package the low-resolution output layers of the source view network and send them to the client, such as using a suitable atlas method. This can have client-side computational benefits (no need to evaluate the first part of the network). On-chip HEVC decoding of video pixel data can often be faster and use less power than using a GPU or even a dedicated neural engine on the client device.
[0208] Thus, instead of providing explicit images, feature values generated by applying appropriate filters may be transmitted. Feature maps are specifically provided for gestures that are not represented by any image / frame included in the audiovisual data stream, and may therefore represent scenes from additional viewing gestures. The feature maps will represent view images from the corresponding viewing gestures but will typically have a much lower resolution and may therefore significantly reduce the data rate of the audiovisual data stream compared to sending the full images.
[0209] As a specific example, an approach can be taken to package the low-resolution output layer of the source view network and send it to the client in an atlas. This can have client-side computational benefits (no need to evaluate the first part of the network). Additionally, on-chip HEVC decoding of the video pixel data can often be faster and use less power than using a GPU or even a dedicated neural engine on the client device.
[0210] Thus, in some embodiments, the audiovisual data stream may include at least one neural network feature map for an image in a viewing posture different from the viewing posture of the plurality of images.
[0211] Then, the neural network controller 405 may be arranged to extract the feature values and configure the neural network using at least one feature map. Specifically, the values of the nodes of the layer for which the feature map is provided may be set to the feature values indicated in the feature map.
[0212] Client neural network 403 may then execute the second portion of the neural network.
[0213] This approach is generally beneficial for neural network based view synthesis operations.
[0214] The data stream generator 305 may be arranged to generate the audiovisual data stream according to any suitable data format, structure or standard.In many embodiments, the audiovisual data stream may be generated according to a video data structure comprising video data and meta or auxiliary data which may include coefficient data and / or spatial data.
[0215] In many embodiments, the data stream generator 305 may be arranged to structure the coefficient data to fit the video data and structure.
[0216] Specifically, inter-frame prediction can be used to encode video data (including image data of frames in an audiovisual data stream). Video encoding can adopt a group of pictures (GOP) structure. In such a GOP structure, frames are divided into GOPs, where each GOP is a collection of frames that can be decoded without the need for information from pictures / frames belonging to another GOP. Within a given GOP, inter-frame prediction and relative encoding are based only on other frames that are part of the same GOP, not on any frames of another GOP. Therefore, inter-frames are encoded only based on other frames of the same GOP. Each GOP will typically include at least one intra-frame. The GOP structure can reflect the arrangement of inter-frames and intra-frames for a group of frames.
[0217] In some embodiments, the data stream generator 305 can be arranged to structure at least two of the image data, scene data, and coefficient data that share a GOP structure. Specifically, the data stream generator 305 can be arranged to structure the scene data and / or coefficient data to use the same GOP structure as the image / video frame.
[0218] For example, a new set of coefficients may be provided for each GOP structure. For example, a set of coefficients may be provided for an intra-coded frame as a set of intra-coded coefficients. Coefficients may then be further provided for an inter-coded frame, wherein the coefficients are also inter-coded based on the coefficients provided for other frames, and may specifically be encoded as relative values relative to coefficients from the intra-coded coefficients provided for the intra-coded frame.
[0219] This approach may allow for an efficient and practical method for transmitting coefficients.
[0220] An artificial neural network suitable for implementing a view synthesis neural network may be a network of nodes arranged hierarchically and wherein each node holds a node value. Figure 7 An example of a portion of an artificial neural network is illustrated.
[0221] The node value of a given node can be calculated to include the contribution of some or usually all nodes from the previous layer of the artificial neural network. Specifically, the node value of a node can be calculated as the weighted sum of the node values output by all nodes of the previous layer, and thus the weight / coefficient can be determined for the node (through training). Typically, a bias can be added, and the result can be subjected to an activation function. The activation function provides the necessary parts of each neuron by generally providing nonlinearity. This nonlinearity and activation function provide significant effects in the learning and adaptation process of the neural network. Therefore, the node value is generated based on the node value of the previous layer.
[0222] The artificial neural network may specifically include an input layer 701, which includes a plurality of nodes receiving input data values for the artificial neural network. Therefore, the node values of the nodes of the input layer may generally be directly the input data values of the artificial neural network, and therefore may not be calculated from other node values.
[0223] The artificial neural network may also include none, one or more hidden layers 703 or processing layers. For each of these layers, the node values are typically generated according to the node values of the nodes of the previous layer and specifically followed by a weighted combination of an activation function and an added bias (such as sigmoid, ReLU or Tanh functions may be applied).
[0224] Specifically, Figure 8 As shown, each node (which may also be referred to as a neuron) can receive input values (from nodes in previous layers) and thereby compute a node value as a function of these values. Typically, this involves first generating a value that is a linear combination of the input values, where each of these input values is weighted by a weight / coefficient:
[0225]
[0226] Among them, w refers to the weight, x refers to the node of the previous layer, and n refers to the index of different nodes in the previous layer.
[0227] In the described methods, coefficients / weights of linear combinations of input values and / or values of nodes of previous layers may be determined by training. The coefficients / weights determined by the training may then be described by coefficient data included in the audiovisual data stream. In many embodiments, bias values (offsets) may be determined for one or more of the nodes, and in many embodiments, data describing the bias values may also be included in the audiovisual data stream.
[0228] An activation function can then be applied to the resulting combination. For example, the node value 1 can be determined as:
[0229] l=f(k)
[0230] The function may be, for example, a rectified linear unit function as described in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (PMLR 15:315-323, 2011) by Xavier Glorot, Antoine Bordes, Yoshua Bengio:
[0231] f(k)=ReLU(k)=max(0,k)
[0232] Other commonly used functions include sigmoid function or tanh function. In many embodiments, multiple functions can be used to calculate node output or value. For example, both ReLU and Sigmoid functions can be combined using activation functions such as the following:
[0233] f(k)=ReLU(k)+σ(k)
[0234] Such operations can be performed by every node of the artificial neural network (usually except the input nodes).
[0235] The artificial neural network also includes an output layer 705, which provides the output from the artificial neural network, i.e., the output data of the artificial neural network are the node values of the output layer. For the hidden / processing layer, the output node values are generated by a function of the node values of the previous layer. However, compared to the hidden / processing layer, where the node values are generally not accessible or further usable, the node values of the output layer are accessible and provide the results of the operation of the artificial neural network.
[0236] Many different network structures and toolkits have been developed for artificial neural networks, and in many embodiments, artificial neural networks can be based on adapting and customizing such networks. An example of a network architecture that can be suitable for the above applications is the long short-term memory (LSTM) [Sepp Hochsreiter, which is described in Hochreiter, Sepp and Jürgen Schmidhuber, "Long short-term memory" (Neural computation 9.8 (1997): 1735-1780)]
[0237] LSTM is an architecture for classification and regression of time-domain signals using recurrent causality or bidirectional evaluation and has been successfully applied to audio signals.
[0238]
[0239] Where * represents matrix multiplication, o represents Hadamard product, x is the input vector, h t-1 represents the output vector of the previous time step, W, V, U are the network weights, and b is the bias vector.
[0240] In theory, classical (or "vanilla") artificial neural networks can track arbitrary long-term dependencies in input sequences. The problem with vanilla artificial neural networks is computational (or practical) in nature: when training a vanilla artificial neural network using backpropagation, the backpropagated long-term gradients can "vanish" (i.e., they can go to zero) or "explode" (i.e., they can go to infinity) because the process involves computations using finite precision numbers. Artificial neural networks using LSTM cells partially solve the vanishing gradient problem because LSTM cells allow gradients to flow unchanged as well. However, LSTM networks can still suffer from the exploding gradient problem.
[0241] In some cases, the artificial neural network may also be arranged to include additional contributions that allow the artificial neural network to be dynamically adapted or customized for a particular desired property or characteristic of the generated output. For example, a set of values may be provided to adapt the artificial neural network. These values may be included by providing contributions to some nodes of the artificial neural network. These nodes may specifically be input nodes, but may generally be nodes of a hidden layer or processing layer. Such adaptation values may, for example, be weighted and added as a contribution to a weighted sum / correlation value of a given node.
[0242] The above description relates to a neural network approach that may be applicable to many embodiments and implementations. However, it should be understood that many other types and structures of neural networks may be used. Indeed, many different methods have been and are being developed for generating neural networks, including neural networks that use complex structures and processes different from those described above. The method is not limited to any particular neural network approach, and any suitable method may be used without departing from the invention.
[0243] Artificial neural networks are adapted to a specific purpose by a training process for adapting / tuning / modifying the weights and other parameters (e.g., bias) of the artificial neural network. It should be understood that many different training processes and algorithms are known for training artificial neural networks. Typically, training is based on a large training set, in which a large number of examples of input data are provided to the network. In addition, the output of the artificial neural network is typically compared (directly or indirectly) with an expected or ideal result. A cost function can be generated to reflect the expected results of the training process. In a typical case known as supervised learning, the cost function typically represents the distance between the prediction and the actual situation for specific input data. Based on the cost function, the weights can be changed, and by repeating the process for the modified weights, the artificial neural network can be adapted toward a state where the cost function is minimized.
[0244] In more detail, during the training step, the neural network can have two different information flows from input to output (forward pass) and from output to input (backward pass). In the forward pass, the data is processed by the neural network as described above, and in the backward pass, the weights are updated to minimize the cost function. Typically, this backpropagation follows the gradient direction of the cost function landscape. In other words, by comparing the predicted output with the actual situation of a batch of data input, the direction in which the cost function is minimized and propagated backward can be estimated by updating the weights accordingly. Other methods known for training artificial neural networks include, for example, the Levenberg-Marquardt algorithm, the conjugate gradient method, and Newton's method.
[0245] In the present case, the training may specifically include a training set comprising a potentially large number of pairs of captured audio signals and measured breathing waveform signals. The measured breathing waveform signals may be generated from sensor signals of a sensor arranged to measure a property that depends on lung air volume. Thus, the training of the artificial neural network may be performed using a training set comprising associated audio / speech data / signals and sensor data / signals representing lung air volume measurements during speech.
[0246] In some embodiments, the training data may be an audio signal in a time period corresponding to the processing time interval of the artificial neural network being trained, for example, the number of samples in the training audio signal may correspond to the number of samples corresponding to the input nodes of the (one or more) artificial neural networks being trained. Thus, each training example may correspond to one operation of the (one or more) artificial neural networks being trained. However, typically, a batch of training samples is considered for each step to speed up the training process. In addition, many updates to the gradient descent may also speed up convergence or avoid local minima in the cost function landscape.
[0247] The device(s) may in particular be implemented in one or more appropriately programmed processors. For example, the neural network may be implemented in one or more such appropriately programmed processors. Different functional blocks and in particular the artificial neural network may be implemented in separate processors and / or may be implemented, for example, in the same processor. Examples of suitable processors are provided below.
[0248] Fig. 9is a block diagram illustrating an example processor 900 according to an embodiment of the present disclosure. The processor 900 may be used to implement one or more processors that implement an apparatus or elements thereof as previously described (particularly including one or more artificial neural networks). The processor 900 may be any suitable processor type, including but not limited to a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable array (FPGA) (where the FPGA has been programmed to form a processor), a graphics processing unit (GPU), an application specific integrated circuit (ASIC) (where the ASIC has been designed to form a processor), or a combination thereof.
[0249] Processor 900 may include one or more cores 902. Core 902 may include one or more arithmetic logic units (ALUs) 904. In some embodiments, core 902 may include a floating point logic unit (FPLU) 906 and / or a digital signal processing unit (DSPU) 908 in addition to or instead of ALU 904.
[0250] The processor 900 may include one or more registers 912 that are communicatively coupled to the core 902. The registers 912 may be implemented using dedicated logic gates (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 912 may be implemented using static memory. The registers may provide data, instructions, and addresses to the core 902.
[0251] In some embodiments, the processor 900 may include one or more levels of cache memory 910 communicatively coupled to the core 902. The cache memory 910 may provide computer readable instructions for execution to the core 902. The cache memory 910 may provide data for processing by the core 902. In some embodiments, the computer readable instructions may have been provided to the cache memory 910 by a local memory (e.g., a local memory attached to the external bus 916). The cache memory 910 may be implemented using any suitable cache memory type, for example, a metal oxide semiconductor (MOS) memory, such as a static random access memory (SRAM), a dynamic random access memory (DRAM), and / or any other suitable memory technology.
[0252] The processor 900 may include a controller 914 that may control inputs to the processor 900 from other processors and / or components included in the system and / or outputs from the processor 900 to other processors and / or components included in the system. The controller 914 may control data paths in the ALU 904, FPLU 906, and / or DSPU 908. The controller 914 may be implemented as one or more state machines, data paths, and / or dedicated control logic. The gates of the controller 914 may be implemented as independent gates, FPGAs, ASICs, or any other suitable technology.
[0253] Registers 912 and cache 910 may communicate with controller 914 and core 902 via internal connections 920A, 920B, 920C, and 920D. The internal connections may be implemented as buses, multiplexers, crossbar switches, and / or any other suitable connection technology.
[0254] Input and output for processor 900 may be provided via bus 916, which may include one or more conductive lines. Bus 916 may be communicatively coupled to one or more components of processor 900, such as controller 914, cache 910, and / or registers 912. Bus 916 may be coupled to one or more components of the system.
[0255] The bus 916 may be coupled to one or more external memories. The external memory may include a read-only memory (ROM) 932. The ROM 932 may be a mask ROM, an electronically programmable read-only memory (EPROM), or any other suitable technology. The external memory may include a random access memory (RAM) 933. The RAM 933 may be a static RAM, a battery-powered static RAM, a dynamic RAM (DRAM), or any other suitable technology. The external memory may include an electrically erasable programmable read-only memory (EEPROM) 935. The external memory may include a flash memory 934. The external memory may include a magnetic storage device, such as a disk 936. In some embodiments, the external memory may be included in the system.
[0256] The present invention can be implemented in any suitable form including any combination of hardware, software, firmware or these projects.Optionally, the present invention can be implemented at least in part as computer software running on one or more data processors and / or digital signal processors.The elements and parts of an embodiment of the present invention can be implemented physically, functionally and logically in any suitable manner.In fact, the function can be implemented in a single unit, in a plurality of units, or be implemented as the part of other functional units.So, the present invention can be implemented in a single unit, or can be distributed between different units, circuits and processors physically and functionally.
[0257] According to standard terminology in the art, the term pixel can be used to refer to properties associated with a pixel, such as light intensity, depth, position of the part / element of the scene represented by the pixel. For example, the depth of a pixel or pixel depth can be understood to refer to the depth of an object represented by the pixel. Similarly, the brightness of a pixel or pixel brightness can be understood to refer to the brightness of an object represented by the pixel.
[0258] Although the present invention has been described in conjunction with some embodiments, the present invention is not intended to be limited to the specific forms set forth herein. On the contrary, the scope of the present invention is limited only by the claims. In addition, although features may be described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined according to the present invention. In the claims, the term "comprising" does not exclude the presence of other elements or steps.
[0259] In addition, although listed separately, multiple modules, elements, circuits or method steps can be implemented by, for example, a single circuit, unit or processor. In addition, although individual features can be included in different claims, these features may also be advantageously combined, and the inclusion of features in different claims does not mean that the combination of features is not feasible and / or advantageous. Moreover, the inclusion of features in a category of claims does not mean that the features are limited to this category, but indicates that the features are equally applicable to other categories of claims where appropriate. In addition, the order of features in the claims does not mean that the features must work in any particular order, and in particular, the order of the various steps in the method claims does not mean that the steps must be performed in this order. On the contrary, the steps can be performed in any suitable order. In addition, singular references do not exclude multiple. Therefore, references to "one", "one", "first", "second", etc. do not exclude multiple. The figure marks provided in the claims are only used to clarify the examples and should not be interpreted as limiting the scope of the claims in any way.
Claims
1. A device comprising: a first receiver (301) arranged to receive a plurality of images of a three-dimensional scene captured from different viewing postures; a second receiver (303) arranged to receive three-dimensional spatial data for the scene; a view synthesis neural network (307) arranged to generate view-shifted images of the scene for different viewing postures based on the plurality of images and the three-dimensional spatial data; a neural network trainer (309) arranged to train the view synthesis neural network (307) based on images of the scene for different viewing poses; A generator (305) arranged to generate an audiovisual data stream, the audiovisual data stream comprising: image data of at least some of the plurality of images; scene data representing the three-dimensional spatial data; Coefficient data describing coefficients of the view synthesis neural network after training.
2. The device according to claim 1, wherein: The first receiver (301) is arranged to receive a set of input video sequences, the set of input video sequences representing views of the three-dimensional scene from the different viewing postures, and the plurality of images are frames of the set of input video sequences; And among them, The neural network trainer (309) is arranged to dynamically train the view synthesis neural network so that coefficients of the view synthesis neural network vary over time; and The generator (305) is arranged to generate the audio-visual data stream to include frames from the set of input video sequences and time-varying coefficient data describing time-varying coefficients of the view synthesis neural network after training.
3. An apparatus according to any preceding claim, wherein: The neural network trainer (309) is arranged to train the view synthesis neural network (307) using a set of training images, the set of training images comprising a first set of reference images for viewing postures not represented by images in the at least some of the plurality of images.
4. The device according to claim 3, wherein: The first set of reference images comprises at least one image selected from the group consisting of: generating a reference image by non-neural network view shifting using the plurality of images; A reference image generated according to a visual scene model of the scene.
5. The device according to claim 3 or 4, wherein: The set of training images comprises a second set of reference images comprising images from the plurality of images, and the neural network trainer (309) is arranged to apply different weights to the first set of reference images and the second set of reference images.
6. An apparatus according to any preceding claim, wherein: The neural network trainer (309) is arranged to encode and decode at least some of the plurality of images before at least some of the plurality of images are provided to the view synthesis neural network.
7. An apparatus according to any preceding claim, wherein: The neural network trainer (309) is arranged to encode and decode the three dimensional spatial data before the three dimensional spatial data is provided to the view synthesis neural network.
8. An apparatus according to any preceding claim, wherein: The neural network trainer (309) is arranged to initialize the view synthesis neural network with a set of default coefficients and to train the view synthesis neural network to determine modified coefficients for the view synthesis neural network (307); and wherein the generator (305) is arranged to include at least some of the modified coefficients in the audiovisual data stream.
9. The device according to claim 8, wherein: The generator (305) is arranged to select a subset of coefficients for transmission in response to a difference between the modified coefficients and the default coefficients.
10. An apparatus comprising: A receiver (401) arranged to receive an audiovisual data stream, the audiovisual data stream comprising: image data representing a plurality of images of a three-dimensional scene captured from different viewing postures, three-dimensional spatial data of the scene, and Coefficient data describing coefficients of a view synthesis neural network; The view synthesis neural network (403) is arranged to generate view-shifted images of the scene for different viewing postures based on the plurality of images and the three-dimensional spatial data; and A neural network controller (405) is used to set coefficients of the neural network in response to the coefficient data.
11. The device according to claim 10, wherein: The audio-visual data stream comprises a set of video sequences, the set of video sequences comprising a plurality of frames representing views of the three-dimensional scene from the different viewing postures, and the plurality of images are frames of the plurality of frames; and the coefficient data comprises time-varying coefficient data describing time-varying coefficients of the view synthesis neural network (403); and the neural network controller (405) is arranged to change the coefficients of the view synthesis neural network (403) in response to the time-varying coefficient data.
12. The device according to claim 11, wherein: The neural network controller (405) is arranged to: determine interpolation coefficient values for at least one time instant, for which coefficient data is not included in the audio-visual data stream, the interpolation coefficient values being determined based on coefficient values of the time-varying coefficient data; and set the coefficients of the view synthesis neural network (403) to the interpolation coefficient values for the at least one time instant.
13. The device according to any one of claims 10 to 12, wherein: The audio-visual data stream comprises at least one neural network feature map for images at a viewing posture different from a viewing posture of the plurality of images, and the neural network controller (405) is arranged to configure the view synthesis neural network (403) using the at least one feature map.
14. The device according to any one of the preceding claims 10 to 13, wherein The neural network controller (405) is arranged to initialize the view synthesis neural network (403) with a set of default coefficients and to overwrite the default coefficients with coefficients from the audiovisual data stream.
15. The device according to claim 14, wherein: The neural network controller (403) is arranged to select the set of default coefficients from a plurality of sets of default coefficients in response to the plurality of images.
16. An apparatus according to any preceding claim, wherein: At least two of the image data, the scene data and the coefficient data share a group of pictures (GOP) structure in the audiovisual data stream.
17. An apparatus according to any preceding claim, wherein: The view synthesis neural network (307, 403) comprises a plurality of tunable layers and a plurality of non-tunable layers, and the audio-visual data stream comprises coefficient data only for the tunable layers.
18. An apparatus according to any preceding claim, wherein: The three-dimensional space data includes video point cloud data for the scene.
19. An audio-visual data stream comprising: image data representing a plurality of images of a three-dimensional scene captured from different viewing postures, three-dimensional spatial data of the scene, and Coefficient data describing coefficients of a view synthesis neural network for generating view-shifted images of the scene for different viewing postures based on the multiple images and the three-dimensional spatial data.
20. An audiovisual data distribution system comprising an apparatus according to any one of claims 1 to 9 and an apparatus according to any one of claims 10-18.
21. A method for generating an audiovisual data stream, the method comprising: receiving a plurality of images of a three-dimensional scene captured from different viewing postures; Receiving three-dimensional spatial data of the scene; A view synthesis neural network (307) generates view-shifted images of the scene for different viewing postures based on the multiple images and the three-dimensional spatial data; training the view synthesis neural network based on images of the scene for different viewing poses (307); and Generate an audio-visual data stream, the audio-visual data stream comprising: image data of at least some of the plurality of images, scene data representing the three-dimensional spatial data, and Coefficient data describing coefficients of the view synthesis neural network (307) after training.
22. A method comprising: Receiving an audio-visual data stream, the audio-visual data stream comprising: image data representing a plurality of images of a three-dimensional scene captured from different viewing postures, three-dimensional spatial data of the scene, and Coefficient data describing coefficients of a view synthesis neural network; A view synthesis neural network (403) generates view-shifted images of the scene for different viewing postures based on the multiple images and the three-dimensional spatial data; and Coefficients of the view synthesis neural network (403) are set in response to the coefficient data.
23. A computer program product comprising computer program code means adapted to perform all the steps of claim 20 or 21 when said program is run on a computer.
