Neural coding and decoding method and device

Abstract

A method of image encoding comprises the steps of training, for a respective image, a respective neural network to output pixel data of the image, and outputting the trained neural network as an encoded representation of the image; meanwhile, a method of image decoding includes the steps of receiving a corresponding neural network trained to output pixel data of an image, and prompting the neural network to output the pixel data of the image.

Description

Technical Field

[0001] The present invention relates to a neural encoding and decoding method and device. Background Art

[0002] The "background" description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent it is described in this background section and insofar as it may not qualify as prior art at the time of filing, is neither explicitly nor implicitly admitted to be prior art against the present invention.

[0003] Conventional video encoders and decoders use one or more of a variety of methods to remove redundant image elements from source images, such as by quantization (to intelligently reduce the number of values ​​to be transmitted), by removing high frequency components (which may have limited visual impact), and by removing spatial and / or temporal redundant components (e.g., repeating blocks of values ​​within an image or corresponding values ​​between consecutive images, optionally after accounting for inter-image motion). These methods can be seen in various well-known encoding techniques such as GIF, JPEG, and MPEG images and videos.

[0004] However, there is a continuing desire to provide more efficient image and video encoding techniques, and the present application seeks to address or alleviate this desire. Summary of the invention

[0005] In a first embodiment, a method of encoding an image is provided according to claim 1.

[0006] In a first example, a method of style encoding is provided according to claim 12.

[0007] In another example, according to claim 14, a method of image decoding is provided.

[0008] In another example, a method of style decoding is provided according to claim 16.

[0009] In another example, an image encoder is provided according to claim 19.

[0010] In another example, a non-transitory machine-readable medium including a representation of one or more images is provided according to claim 20.

[0011] In another example, an image decoder is provided according to claim 21.

[0012] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary rather than restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A more complete understanding of the present disclosure and many of its attendant advantages will be readily obtained as the present disclosure and many of its attendant advantages become better understood by referring to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0014] Figure 1 is a schematic diagram of an entertainment device that can be configured as an encoder and / or a decoder according to an embodiment of the present specification.

[0015] Figure 2 is a schematic diagram of a neural network according to an embodiment of the present specification.

[0016] Figure 3 is a schematic diagram of a neural network according to an embodiment of the present specification.

[0017] Figure 4 is a flowchart of an image encoding method according to an embodiment of the present specification.

[0018] Figure 5 is a flowchart of an image decoding method according to an embodiment of the present specification. DETAILED DESCRIPTION

[0019] Referring now to the drawings, in which like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 In the example of the entertainment system 10 is a computer or a console, such as PlayStation (PS5)).

[0020] The entertainment system 10 includes a central processor 20. This can be a single core or multi-core processor, for example including eight cores as in the PS5. The entertainment system also includes a graphics processing unit or GPU 30. The GPU can be physically separate from the CPU, or integrated with the CPU as a system on a chip (SoC) as in the PS5.

[0021] The entertainment device also includes RAM 40, and may have separate RAM for each of the CPU and GPU, or shared RAM as in the PS5. The or each RAM may be physically separate, or integrated as part of the SoC as in the PS5. Further storage is provided by a disk 50, either as an external or internal hard drive, or as an external solid state drive, or as an internal solid state drive as in the PS5.

[0022] The entertainment device may be suitably connected via one or more data ports 60 (such as a USB port, port, port, Port or similar port) to send or receive data. It can also optionally receive data via optical drive 70.

[0023] Audio / video output from the entertainment device is typically provided through one or more A / V ports 90 or through one or more of the wired or wireless data ports 60 .

[0024] An example of a device for displaying images output by the entertainment system is a head mounted display “HMD” 120 (such as a PlayStation VR 2 “PSVR2”) worn by the user 1 .

[0025] Where components are not integrated, they may be connected by dedicated data links or via bus 100 as appropriate.

[0026] Interaction with the system is typically provided using one or more handheld controllers (130, 130A), such as a controller (130) in the case of a PS5, and / or one or more VR controllers (130A-L, R) in the case of an HMD.

[0027] Encoder

[0028] In the embodiments of the present specification, the entertainment system 10 is an example of a coding system operable to implement the methods and techniques described herein. The coding system can be used to encode and / or decode images and / or videos according to the methods and techniques described herein.

[0029] In an embodiment of the present specification, the encoding method includes training a machine learning system (such as a neural network).

[0030] An example neural network would be a fully connected multilayer perceptron, i.e., a fully connected multilayer network with one or more hidden layers. Typically, such a neural network operates as a feedforward neural network, trained using a backpropagation algorithm to approximate a target output in response to a corresponding input. More generally, a neural network may be implemented as a fully connected neural network, a deep neural network (DNN), a multilayer perceptron (MLP), a feedforward artificial neural network (ANN), or a convolutional neural network (CNN or ConvNet).

[0031] In particular, in the embodiments of the present specification, each image / frame is encoded as a neural network. That is, the corresponding neural network is trained to generate pixel data corresponding to an image. The neural network is then an implicit neural representation of the source content of the image / frame.

[0032] Then, the corresponding neural network sequences are stored, transmitted or streamed instead of the image / frame data or its conventional encoded version.

[0033] The method has a high efficiency range because the neural network can learn how to best represent a given image within the weights that comprise it, and the weights are then effectively a compressed representation of the image. The image bandwidth can then be controlled by the number of nodes within the neural network used to encode the image, which in turn determines the number of weights available to the neural network.

[0034] The degree of compression is therefore controlled by the number of nodes (typically hidden nodes) within the neural network used to represent the corresponding image, and each neural network learns the best representation of the image for the available bandwidth provided by the network's weights.

[0035] Now go to Figure 2 , the example neural network 200 includes an input node layer 210, an output node layer 240, and one or more hidden layers 220, 230. The hidden layers may include any suitable number of nodes. Between each layer is typically a fully connected set of weights 215, 225, 235; that is, there is a weight between each node in adjacent layers. However, optionally, the network need not be fully connected in this manner, and therefore some weights may not be included.

[0036] In the embodiments of the present specification, the two input nodes (x, y) indicate the pixel position within the image to be encoded, and the three output nodes (R, G, B) indicate the color value of the pixel at the x, y pixel position. It should be understood that RGB is an example color scheme, and any suitable color (or grayscale) representation (such as, for example, YUV or HSV) may be used.

[0037] A neural network is trained on a particular image (e.g., an image or image frame of a video) using the RGB values ​​of the image at different x, y positions. Training can continue until an error criterion is met.

[0038] Error criteria may include some or all of the following:

[0039] i. The total error of all image pixels is below a threshold amount (indicating that the image accuracy meets a predefined quality / accuracy level).

[0040] - Alternatively, the total error of a subset of pixels is below a single threshold amount (simply indicating that a representative sample of pixels meets a predefined quality / accuracy level).

[0041] ii. The average error of all pixels or a subset of pixels is below a threshold amount (it will be appreciated that the average is a similar measure to the total in order to meet predefined quality / accuracy levels).

[0042] iii. No more than N image pixels have errors above a threshold amount (where N indicates the maximum acceptable number of pixels in the image with significant errors - such pixels may optionally be subsequently corrected using error concealment techniques at the decoder).

[0043] iv. After M training iterations, the total error or mean error has not changed by a threshold amount (i.e., the neural network may have achieved its best available result for the current training process, or subsequent training iterations will represent a sufficiently small return on the computational investment that they should not continue.

[0044] Once trained, the neural network includes an internal representation of the image within its weights.

[0045] Assuming that the architecture / layout of the neural network and the decoder are known, only the weight values ​​of the neural network can be sent to the decoder in a predetermined order.

[0046] Optionally, during training, weight values ​​can be constrained to be values ​​suitable for transmission. For example, typically weights are in the range 0 to 1; however, the system can be forced to use weight values ​​that are fractions of 256 or 65536 during training (e.g., by selecting the fraction closest to the calculated weight change); the weights can then be accurately represented using one or two bytes each. Other bitwise plantations of values ​​can be considered, or one or more quantization schemes can be used to select the P most common values ​​within the network (or between corresponding layers).

[0047] The above approach is intended to enable the network to perform correctly / as expected using weight values ​​that can be efficiently sent to the decoder.

[0048] With this approach, modifications to the training scheme can also be considered; for example, the network can first be trained conventionally, then the weight values ​​can be constrained to fractions such as those described above or to values ​​corresponding to the quantization scheme, and then only a subset of the network (e.g., within a layer) can be given supplementary training to compensate for small changes in weight values ​​imposed throughout the network by this method; optionally, the layer can be sent with a different fidelity of value. Or, for example, most layers can be trained using fractions of 256, with one layer (e.g., the final set of weights leading to the output node) trained using fractions of 65535.

[0049] In any case, the weights of the neural network can then be sent from the encoder (or a server or other transmitter with access to the encoder's stored results) to the decoder, for example as a data stream or as a file, or exported as part of a software product such as a compact disc.

[0050] Decoder

[0051] The decoder then queries the received neural network for each x, y pixel location and stores the output pixel information for that location.

[0052] It will be appreciated that for a typical HD image (1920x1080) this would represent over 2 million queries, and for a UHD image (3840x2160) this would represent over 8 million queries.

[0053] Therefore, in an embodiment of the present disclosure, a decoder includes a graphics processing unit having multiple parallel shaders, each parallel shader being operable to query a corresponding copy of a neural network to generate results in parallel, thereby accelerating the decoding process.

[0054] So, instead of decoding the image in the conventional way, the decoder prompts one or more copies of the neural network to generate image information at the corresponding pixel locations.

[0055] Variants

[0056] I and P Network

[0057] The above method provides a means for encoding an image or a sequence of images (such as a video) by creating a corresponding trained neural network for each image, which outputs pixel information of the image corresponding to each queried pixel position.

[0058] This requires that the neural network build a substantially complete internal representation of the image. The neural network can therefore be thought of as an "I-frame" neural network; that is, a neural network that completely encapsulates the image it represents. The neural network is able to do this very efficiently, but there is room to improve this further over sequences of images.

[0059] Now refer to Figure 3 , which is consistent with Figure 2 In a similar manner to FIG. 3 , a variant neural network 300 is shown comprising an input layer 310, an output layer 340 and one or more hidden layers 320, 330. However, in this case, there are three additional inputs R t-1 , G t-1 , B t-1 , which corresponds to the RGB value of the x,y position in the previous image in the image sequence. More specifically, the additional input corresponds to the RGB value of the x,y position in the image generated by the previous neural network, because this is what the decoder will access when prompting the subsequent neural network to generate an output.

[0060] The variant neural network is then trained to output the pixel value R of the current image based on the pixel position and the pixel value at that position in the previous imaget , G t , B t .

[0061] This exploits the assumption that previous images in a sequence of images are usually very similar to the current image. Therefore, in this case, the neural network only needs to learn how to modify the input from the previously decoded / output image to generate the new image, without storing the complete internal representation of the image itself. Therefore, in general the neural network can achieve similar quality results with far fewer weights, which means that the overall size of the neural network can be smaller, and therefore the bandwidth required to send or store the neural network is also smaller.

[0062] This neural network can be thought of as a P-frame neural network; that is, the neural network relies on previous image data to generate its output.

[0063] Optionally, the three additional inputs may be RGB values ​​for pixel locations in a previous image that are different from the pixel locations represented by the input x, y coordinates of the current image being encoded. In particular, the input RGB values ​​may take into account inter-frame motion, and represent the RGB value of the location of the corresponding image element in the previous image at that time, so that it matches the current RGB value at the x, y coordinates well. Inter-frame motion may be limited to whole image translations, or more complex motion within an image, represented, for example, by motion vectors.

[0064] In order to also input the same RGB values ​​at the decoder, the decoder must independently estimate the same inter-frame motion, or the inter-frame motion information is provided as separate data associated with the neural network when it is sent. Alternatively, the neural network can also be trained to output such information by providing such information as target data during training.

[0065] The trained neural network sequence may therefore include a mix of I-frame and P-frame neural networks, with I-frames typically used when there is a scene change, or where the difference between subsequent images is too great for the P-frame neural network to generate an output with an acceptable level of error. P-frames may also optionally be accompanied by information indicating inter-frame motion, or information to enable its calculation at the decoder, where this has been used during training.

[0066] It is also possible to provide R to the I frame neural network t-1 , G t-1 , G t-1Input but using a standard image chosen to improve the efficiency of the I-frame neural network. For example, the normalized prior image may be one of Q images that is an average of a particular type of image; e.g., landscape images, cityscape images, talking head images, etc.; these will not be described in detail, but will include typical features such as a lighter upper half of the image with a particular color range and a darker lower half with a different particular color range for landscape images, a different division of light and color for cityscape images, a general human outline and a general background for talking head images, etc. In this case, Q may be, for example, 8, 16, 32, etc.

[0067] The I-frame neural network must then learn how to modify this generic image into the specific image it is being trained on; because the normalized generic preceding image is only a poor approximation of the current image, this neural network will generally need more weights than a normal P-frame neural network, but because it is only computing differences, it may need fewer weights than a pure I-frame network without input from the generic preceding image, and will therefore encode the image more efficiently.

[0068] Thus, during training, a normalized previous image within the set of Q images that is closest to the image being encoded (e.g., in terms of the sound squared error of pixel values) is selected and used to provide R during training. t-1 , G t-1 , B t-1 The input and the selected image are indicated as metadata along with the trained network when sent to the decoder. The decoder can then select the correct normalized previous image when prompted with the output from the trained network.

[0069] It should also be understood that Q standard images can be generated from the specific content being encoded so that the colors and lighting within the standard images are a better initial match than if a completely generic image were used; these Q standard images can then be encoded using an I-frame neural network or any other conventional technique and sent, for example, at the beginning of a stream or at any event before they are needed.

[0070] In this way, for at least most of the encoded streams, the inputs and outputs of the corresponding neural networks, and potentially their footprint (number of weights), can be the same, making the decoding process simpler in terms of consistently placing and running the neural networks on the GPU.

[0071] Block input and / or output

[0072] It should be appreciated that it would not be efficient for the neural network to have an output node for each pixel of the decoded image, as this would mean that internally there would be multiple weights for each pixel, and it is likely that the neural network is actually much larger than the original image, with many of the weights contributing to individual pixels being highly redundant.

[0073] However, outputting the RGB (or other color scheme) pixel value of a single pixel represents the opposite extreme; it provides efficient use of weights, especially those connected to the output layer, but can therefore cause these rates to be very sensitive to changes in values ​​or restrictions on value choices (such as those discussed previously for more efficient transmission of weight values), and also forces the decoder to implement a large number of queries to generate the entire image.

[0074] Thus, the neural network may optionally be configured to provide pixel values ​​for a larger subset of the image. For example, the neural network may output pixel values ​​for 2×2, 4×4, 8×8, 16×16, 32×32, or 64×64 pixel blocks (e.g., macroblocks or equivalent). Thus, the output layer (240, 340) may include more than Figure 2 or Figure 3 The hidden layer (or the hidden layer next to the output layer 230, 330) may also include fewer nodes, so that the number of weights (235, 335) between the hidden layer and the output layer is roughly similar to the number of weights of the single pixel output version.

[0075] Similarly, and optionally independent of the selection of single or multiple output pixels, the neural network can be configured to have more input values ​​than, for example, just the RGB value of a single pixel at an x,y location. For example, the neural network can have input pixel values ​​of 2×2, 3×3, 4×4, 5×5, 8×8, 16×16, 32×32, or 64×64 pixel blocks. Likewise, the hidden layer (220, 320) adjacent to the input layer (210, 310) can include a fewer total number of intermediate weights (215, 315) roughly similar to the annotation of the single input version.

[0076] Alternatively, the input RGB values ​​may correspond to a lower resolution version of the target image; thus, for example, a 2×2 or 4×4 group of RGB values ​​may represent a downsampled 4×4, 8×8, or 16×16 pixel region of the previous image.

[0077] In the case where the input region of the previous image is larger than the output region of the corresponding image, then the input region provides some context to the neural network when generating the output region values, which can be particularly helpful for some styles (as described elsewhere in this paper), but some of this may be redundant and reduce overall efficiency in the network while potentially improving image quality. Meanwhile, in the case where the input region is the same size as the output region (whether using a downsampled version of the input region or for pixel-by-pixel values ​​of the input region), then the network operates in a similar manner to that previously described, but for multiple pixels within a block.

[0078] It should also be understood that in neural networks that facilitate pixel data in a block-wise manner, the x, y inputs may correspond to block positions rather than pixel positions.

[0079] Stereo Vision

[0080] The neural network may also be trained to output two sets of RGB values ​​for corresponding coordinates in a stereo image pair. In this case (or more generally, where applicable, whether or not for stereo images), the neural network may also receive depth information (e.g., from a corresponding depth map) or disparity information to assist in the internal representation of the stereo scene. In this case, the two sets of RGB values ​​will likely be different because the same location in each image may include different elements due to the disparity / offset of the stereo images.

[0081] So instead, the neural network can be trained to output two sets of RGB values ​​and disparity / offset values, the first set of RGB values ​​corresponding to the input coordinates of one of the image pair (as an arbitrator example, the left side), and the second set corresponding to the position offset from the input coordinates according to the stereo disparity of the other image in the image pair (in this case, the right side). In this way, the neural network can learn to output the same elements of the scene as offsets due to stereo vision, and therefore the RGB values ​​may be very similar.

[0082] These two methods represent different internal representation costs for the process of predicting the color or position of stereoscopic image elements, and different methods may be more effective for different types of content or within specific content segments, respectively.

[0083] Various styles

[0084] Different users may wish to view content in different ways. This may be done for purely aesthetic reasons, or it may be beneficial for a particular class of users. For example, it may be advantageous to provide content in a format that is best viewed by color blind users (e.g., for various degrees of red-green color blindness, or similarly for various degrees of blue-yellow color blindness).

[0085] Aesthetic changes may include differences in contrast and / or color balance, or other effects such as creating a paint, pastel, or line drawing version of the image or some other texture modification, a highly saturated version, a warm or cool color temperature version, etc.

[0086] Conventionally, such modifications are applied by the client device after the source image is decoded. However, in embodiments of the present specification, the style may optionally be incorporated into the neural network rather than applying the style at the decoder.

[0087] Optionally, where it can be assumed that a style persists for a significant portion of the viewing period (e.g. relevant to color blind users, or where the user selects a particular color temperature version in response to settings on the television or properties of their local environment), then a neural network (in any of the forms previously described in this document) can be trained specifically to output pixel values ​​in that style, and the associated neural network sequences can be streamed.

[0088] However, for example in a broadcast scenario where multiple viewers may receive a single stream, it may not be practical to output individually customized streams due to the different needs of the viewers.

[0089] In this case, optionally, the neural network (in any form that has been previously described herein, e.g., fully encapsulating the encoded image, modifying the normalized image, or modifying a previous image, whether outputting a single pixel at a time or multiple pixels at a time) may also have one or more style inputs.

[0090] The style input may take the form of a binary input or a series of flags (e.g. as a one-hot vector) to a corresponding additional input node of the network. The binary or one-hot flag pattern indicates the choice of style. Typically, a style is neutral, i.e., represents the original image. Alternatively or additionally, the style may be represented by a vector, a scalar, or actual text (e.g., converting the text to a vector, optionally as an abstraction of the text).

[0091] The neural network is then trained on multiple versions of the image (i.e., at least two), one for each supported style. Thus, in this case, the neural network may output different RGB values ​​for the same pixel location depending on the style indicated. In this case, for comparable image quality / accuracy, the neural network will be slightly larger than a neural network that encodes a single style or no style at all (i.e., a "neutral" style). Such a neural network can be thought of as analogous to a combination of an I-frame or P-frame network and another P-frame network for each supported style, with weights driven by the style input interacting with other weights to modify the image generated by the rest of the network in response to the input coordinates and optionally the input RGB values.

[0092] This multi-style version of the neural network may then be sent to a decoder in a manner similar to that described elsewhere in this document, and prompted by the decoder to output pixel values ​​in a similar manner, but in this case the relevant style selected by the user (e.g. via a UI) may also be input to the network by the decoder, so that RGB values ​​appropriate to that style are then output.

[0093] Thus, in this approach, styles are not applied to decoded images at the client device; instead, multiple styles are embodied within a neural network representation of the image (or image modification), and the decoder prompts the neural network to output the relevant style type.

[0094] Style parameterization

[0095] In the case of training a neural network on one or more styles as previously described herein, one or more of the styles may optionally also include one or more parameter inputs to control one or more aspects of the style.

[0096] For example, a style that modifies the color temperature of an image may include a time of day parameter so that the color temperature gradually changes from cooler to warmer toward late night, which may help a user sleep. In this case, a neural network may be trained on different color temperatures associated with different time of day parameters.

[0097] Alternatively, a neural network can be trained on warm and cold versions of the image as separate styles, and the parameters can be values ​​representing the relative deviation between the two styles. Thus, in this case, for example, the style inputs that together represent a potential range or spectrum of styles for the output can each be continuous values ​​rather than flags, with the corresponding values ​​indicating their contribution to the output.

[0098] In a similar manner, potentially any style could include a parameter input or continuous style input that indicates the degree / intensity of stylization relative to a neutral version of the image. In this case, the neural network would again be trained accordingly with different parameter input values ​​and different degrees of intensity / stylization in the target output.

[0099] This also allows different styles to be combined by providing a style input that indicates the degree / intensity to which each style is applied. For example, a pencil sketch style can be applied and a sepia palette style applied separately to create a combined effect that itself was not part of the training set of target images.

[0100] Style post-processing

[0101] Optionally, given that the output of a neural network according to the techniques herein is an RGB value (or similar pixel value) for a given input coordinate, a flow according to an embodiment of the present specification may optionally include a neural network with no style / neutral style output, and a separate second neural network that applies one or more styles to the output of the first neural network.

[0102] This second neural network can be trained in a manner similar to the style-based neural networks previously described herein, and acts similarly to a P-frame network, receiving RGB values ​​and at least a first style value and typically coordinate values ​​from the first neural network, and outputting corresponding RGB values ​​for the selected style.

[0103] In this case, the first network can be viewed as a baseline public transmission to clients, while the second neural network can be viewed as an enhancement layer provided to clients requesting a modified image (i.e., a different style).

[0104] Other post-processing

[0105] It will be appreciated that regardless of whether the style is generated by or applied to the output of the first neural network, other more conventional post-processing may optionally be performed on the resulting image.

[0106] Such post-processing may include error concealment, amplification, jitter reduction, etc.

[0107] Overview

[0108] Now refer to Figure 4 In an overview embodiment of the present description, a method of image encoding (eg for still or video images) comprises the following steps.

[0109] In a first step S410, for a corresponding image, a corresponding neural network is trained to output pixel data of the image, as described elsewhere herein.

[0110] In a second step S420, the trained neural network is output as an encoded representation of the image (whether as a single image, or as a stream of consecutive images such as in a video, broadcast and / or file), as described elsewhere herein.

[0111] It will be apparent to those skilled in the art that variations of the above methods corresponding to the operation of the various embodiments of the apparatus described and claimed herein are considered to be within the scope of the present invention, including but not limited to:

[0112] - the respective neural network may comprise an input receiving coordinates of locations in the respective image, and an output providing color values ​​of pixels at those coordinates, as described elsewhere herein;

[0114] - in this example, the corresponding neural network may also include color value inputs for receiving pixels of the cue image for those received coordinates, as described elsewhere herein;

[0115] - in this example, in turn, optionally, the hint image may be one selected from a list consisting of: an image immediately preceding the corresponding image; and an image selected from a set of generic images as being the image most similar to the corresponding image, as described elsewhere herein;

[0116] - similarly, in this example, in turn, alternatively or additionally, optionally, the received color values ​​of pixels of the hint image may be offset from the received coordinates by an amount of pixel offset in response to inter-image motion between the corresponding image and the hint image, as described elsewhere herein;

[0117] - in this example, the corresponding neural network may include an input for receiving coordinates of locations in the corresponding image, and an output for providing color values ​​of a set of pixels at those coordinates, the set being smaller than the entire corresponding image, as described elsewhere herein;

[0118] - in this example, the respective neural network may include inputs for receiving coordinates of locations in the respective images, and outputs for providing color values ​​of pixels in each of the stereoscopic image pair, as described elsewhere herein;

[0119] - The corresponding neural network may also include inputs defining two or more styles, wherein

[0120] At least one style modifies the output pixel data of the corresponding image, as described elsewhere herein

[0121] described;

[0122] - in this example, optionally, the corresponding style modifies the output pixel data to change one or more selected from a list consisting of: color properties associated with the type of display on which the corresponding image is displayed (e.g., to account for differences in color balance or brightness between displays or display types such as a TV versus a head mounted display); color properties responsive to the time of day (e.g., lowering the color temperature later in the evening to help the viewer sleep); color properties responsive to a medical condition of the viewer (e.g., to help users who have color blindness or who would benefit from high contrast images), aesthetic color properties (e.g., sepia or 1950s style highly saturated film effects), and texture properties (e.g., to make the image appear painted or drawn, or have a grainy film quality, or have a cathode ray tube rasterization style), as described elsewhere herein;

[0123] - similarly, in this example, alternatively or additionally, the corresponding neural network may optionally include one or more style inputs (e.g., alone or in combination with one or more other styles supported by the neural network) for indicating the degree or intensity to which the style should be applied to the output pixel data, as described elsewhere herein; and

[0124] -As an alternative to incorporating the style into a single neural network for an image, then for a corresponding image, training another neural network to output style-modified pixel data for that image, the input to the other neural network comprising the output pixel data of the corresponding neural network, and outputting the trained other neural network as a supplemental encoded representation of the image, as described elsewhere herein.

[0125] Separately, it will be appreciated that while the above relates to encoding an image in a respective neural network, optionally with two or more additional styles (one of which may be "original"), or optionally providing a second neural network to apply the two or more additional styles, alternatively, only the style may be encoded independently of the method of encoding or decoding the image.

[0126] Thus, in an embodiment of the present specification, a method of style encoding comprises the following steps: providing a corresponding trained neural network, the trained neural network comprising an input for receiving color values ​​and defining two or more styles for a corresponding image, and an output for providing color values, the neural network having been trained to output the color of the corresponding image in two or more styles, wherein at least one style modifies the output color of the corresponding input color value; and outputting the trained neural network to a decoder for use with a decoded image. Optionally, the corresponding neural network comprises an input for receiving coordinates of locations in the corresponding image, and an output having been trained to provide color values ​​of pixels at those coordinates.

[0127] In this way, a style network can be sent that can apply a style to an existing image (e.g., an image decoded from another network as described elsewhere herein, or an image decoded using any other technique).

[0128] At the decoder, the indicated style may be applied generally to the image, or alternatively in response to the current pixel position. Thus, in an embodiment of the present specification, a method of style decoding comprises the following steps: receiving a corresponding neural network that is trained to output color values ​​in response to input color values ​​and input style values ​​defining two or more styles, at least one of which modifies the output color of the corresponding input color value; inputting the color values ​​of the corresponding image and at least a first style value to the neural network, and receiving the output color values ​​from the neural network; and displaying the corresponding image using the output color values ​​instead of the input color values. Optionally, the corresponding neural network includes an input for receiving coordinates of a position in the corresponding image, and an output that has been trained to provide color values ​​for pixels at those coordinates.

[0129] Now refer to Figure 5 In the overview embodiment of the present specification, the method of image decoding includes the following steps.

[0130] In a first step s510, receiving a corresponding neural network trained to output pixel data of an image, as described elsewhere herein; and

[0131] In a second step s520, the neural network is prompted to output pixel data for the image, as described elsewhere herein.

[0132] It will be apparent to those skilled in the art that variations of the above methods corresponding to the operation of the various embodiments of the apparatus described and claimed herein are considered to be within the scope of the present invention, including but not limited to:

[0133] -The corresponding neural network includes an input for receiving coordinates of locations in the corresponding image, and an output for providing color values ​​of pixels at those coordinates, and prompting the neural network to output pixel data of the image including the coordinates of all locations in the input corresponding image and collating the output as described elsewhere herein.

[0134] Modifications to the decoder corresponding to the modifications described elsewhere herein will be apparent to those skilled in the art; for example, modifications to generate output on a pixel-by-pixel basis or on a group-of-pixels basis; modifications to use color values ​​from a hint image, which hint image may be from a previously generated image or a best matching generic image; modifications to output pixels of corresponding images from a stereo pair; modifications to input one or more style selections; or parameter values ​​associated with one or more style selections; and modifications involving receiving a first neural network that encodes an image and a second neural network that encodes a style change so as to generate a style-modified version of the image, and modifications involving implementing these processes in parallel on a GPU or equivalent processor with parallel processing capabilities so that corresponding instances of the hint neural network can generate individual pixels or groups of pixels as output in parallel to speed up the overall generation of the image.

[0135] It should be understood that the above-described methods may be performed on conventional hardware appropriately adapted to the application by software instructions or by the inclusion or replacement of dedicated hardware.

[0136] Thus, the required adaptation of existing parts of conventional equivalent devices can be implemented in the form of a computer program product comprising processor-implementable instructions stored on a non-transitory machine-readable medium (such as a floppy disk, optical disk, hard disk, solid-state disk, PROM, RAM, flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (application-specific integrated circuit) or FPGA (field programmable gate array) or other configurable circuit suitable for adapting conventional equivalent devices. Separately, such a computer program can be sent via a data signal on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0137] Thus, in the outlined embodiment of the present specification, an image encoder (eg, a server (not shown) or an entertainment device 10 (eg, a Sony PlayStation )) may include the following:

[0138] First, a training processor (e.g., GPU 30 optionally in conjunction with CPU 20) is configured (e.g., via appropriate software instructions) to train a corresponding neural network for a corresponding image to output pixel data for the image, as described elsewhere herein.

[0139] Second, an output processor (e.g., CPU 20 optionally in conjunction with GPU 30) is configured (e.g., via appropriate software instructions) to output (e.g., via data port 60, SSD 50, optical drive 70, or A / V port 90) the trained neural network as an encoded representation of an image (e.g., an individual image or image frame as part of a sequence such as a video), as described elsewhere herein.

[0140] Examples of embodiments of the present disclosure that implement the methods and techniques described herein (eg, by using appropriate software instructions) are contemplated to be within the scope of this application.

[0141] The resulting output can be a non-transitory machine-readable medium (such as an HDD or SSD record, or an optical drive record) that includes representations of one or more images formed by a corresponding neural network for the one or more images, each image trained to output pixel data for the corresponding image (e.g., an individual image or an image frame as part of a sequence such as a video), as described elsewhere herein.

[0142] The output may be distributed on a medium, either broadcast or streamed as appropriate.

[0143] Meanwhile, in another general embodiment of the present specification, an image decoder, such as a server (not shown) or an entertainment device 10 (such as a Sony ) includes the following:

[0144] First, a receiver (e.g., CPU 20 optionally in conjunction with GPU 30) is configured (e.g., via appropriate software instructions) to receive (via data port 60, SSD 50, optical drive 70, or A / V port 90) a corresponding neural network trained to output pixel data of an image, as described elsewhere herein.

[0145] Second, an image generation processor (e.g., GPU 30 optionally in conjunction with CPU 20) is configured (e.g., via appropriate software instructions) to prompt the neural network to output pixel data for the image, as described elsewhere herein.

[0146] Again, examples of embodiments of the present disclosure that implement the methods and techniques described herein (e.g., by using appropriate software instructions) are contemplated to be within the scope of this application, including, but not limited to:

[0147] - the image generation processor is a graphics processing unit having multiple parallel processing units, and

[0148] The parallel instances of the neural network are configured to output pixel data for the image in parallel, as described elsewhere herein.

[0149] The foregoing discussion discloses and describes only exemplary embodiments of the present invention. As will be appreciated by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the disclosure of the present invention is intended to be illustrative, rather than limiting, of the scope of the present invention and the claims. The present disclosure, including any readily discernible embodiment taught herein, Variants , in part, defines the scope of the preceding claim terms so that no inventive subject matter is dedicated to the public.

Claims

1. An image encoding method comprising the following steps: For a corresponding image, a corresponding neural network is trained to output pixel data of the image; as well as The trained neural network is output as an encoded representation of the image.

2. The method according to claim 1, wherein: The corresponding neural network includes an input for receiving the coordinates of locations in the corresponding image, and an output for providing the color values ​​of the pixels at those coordinates.

3. The method according to claim 2, wherein: The corresponding neural network also includes an input for receiving the color values ​​of the pixels of the hint image for those received coordinates.

4. The method according to claim 3, wherein: The prompt image is one selected from a list consisting of: i. the image immediately preceding the corresponding image; and ii. Selecting an image from a set of common images as the image most similar to the corresponding image.

5. The method according to claim 3, wherein: The received color values ​​of the pixels of the hint image are offset from the received coordinates by an amount of pixel offset in response to inter-image motion between the corresponding image and the hint image.

6. The method according to any one of claims 2 to 5, wherein: The corresponding neural network includes an input for receiving coordinates of locations in the corresponding image, and an output for providing color values ​​of a set of pixels at those coordinates, the set being smaller than the entire corresponding image.

7. The method according to any one of claims 2 to 5, wherein: The respective neural network comprises an input for receiving coordinates of locations in the respective images, and an output for providing color values ​​of pixels in each of the stereoscopic image pair.

8. A method according to any one of the preceding claims, wherein: the corresponding neural network has been trained to output pixel data of the corresponding image in two or more styles, at least one of which modifies the output pixel data of the corresponding image; and The corresponding neural network includes inputs defining the two or more styles.

9. The method according to claim 8, wherein: The corresponding style modifies the output pixel data to change one or more selected from the list consisting of: i. a color attribute associated with the type of display on which the corresponding image is displayed; ii. Color attributes responsive to time of day; iii. Color attributes responsive to the medical condition of the viewer; iv. Aesthetic color attributes; and v. Texture attributes.

10. The method according to claim 8, wherein: The corresponding neural network may include one or more style inputs indicating the degree or intensity to which the style should be applied to the output pixel data.

11. The method according to any one of claims 1 to 7, wherein: For the corresponding image, training another neural network to output style-modified pixel data of the image, the input to the another neural network comprising the output pixel data of the corresponding neural network; as well as The trained another neural network is output as a complementary encoded representation of the image.

12. A method for style encoding, comprising the following steps: providing a corresponding trained neural network, the trained neural network comprising an input for receiving color values ​​and defining two or more styles for a corresponding image, and an output for providing the color values, the neural network has been trained to output colors of the corresponding image in the two or more styles, at least one of which modifies the output color of a corresponding input color value; and The trained neural network is output to a decoder for use with a decoded image.

13. The method according to claim 12, wherein: The corresponding neural network comprises an input for receiving the coordinates of locations in the corresponding image, and an output that has been trained to provide the color values ​​of the pixels at those coordinates.

14. A method for image decoding, comprising the following steps: receiving a corresponding neural network trained to output pixel data of an image; as well as Prompt the neural network to output the pixel data of the image.

15. The method according to claim 14, wherein The corresponding neural network comprises an input for receiving coordinates of locations in the corresponding image, and an output for providing color values ​​of pixels at those coordinates; and The pixel data of the neural network output image includes the coordinates of all positions in the input corresponding image and is sorted for output.

16. A method for style decoding, comprising the following steps: receiving a corresponding neural network trained to output a color value in response to an input color value and an input style value defining two or more styles, at least one of which modifies an output color of the corresponding input color value; inputting color values ​​of a corresponding image and at least a first style value to the neural network, and receiving output color values ​​from the neural network; as well as The corresponding image is displayed using the output color value instead of the input color value.

17. The method according to claim 16, wherein: The respective neural network comprises an input for receiving coordinates of locations in the respective image, and an output that has been trained to provide color values ​​for pixels at those coordinates.

18. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method of any preceding claim.

19. An image encoder, comprising: A training processor configured to train a corresponding neural network for a corresponding image to output pixel data of the image; as well as An output processor is configured to output the trained neural network as an encoded representation of the image.

20. A non-transitory machine-readable medium comprising representations of one or more images formed by respective neural networks for the one or more images, each neural network trained to output pixel data for the respective image.

21. An image decoder comprising: a receiver configured to receive a corresponding neural network trained to output pixel data of an image; as well as An image generation processor is configured to prompt the neural network to output the pixel data of the image.

22. The image decoder according to claim 21, wherein The image generation processor is a graphics processing unit having a plurality of parallel processing units and is configured to prompt parallel instances of the neural network to output pixel data of the image in parallel.

Images