Neural coding and decoding method and apparatus

Abstract

To provide a neural coding and decoding method, an image coding apparatus, an image decoding apparatus, and a non-transitory machine-readable medium, for more efficient image and video encoding techniques.SOLUTION: A method of image coding comprises the steps of: training a neural network to output pixel data for an image; and outputting the trained neural network as an encoded representation of the image. A method of image decoding comprises the steps of: receiving a neural network trained to output pixel data for an image; and prompting the trained neural network to output the pixel data for the image.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a neural encoding and decoding method and apparatus. [Background technology]

[0002] The "Background" discussion provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors, to the extent that it is described in this Background section, as well as aspects of this specification that may not qualify as prior art at the time of filing, are not admitted expressly or impliedly as prior art against the present invention.

[0003] Conventional video encoders and decoders use one or more of a variety of methods to remove redundant image elements from a source image, such as, for example, quantization (to intelligently reduce the number of values ​​to transmit), removal of high frequency components (which may have limited visual impact), and removal of spatially and / or temporally redundant components (e.g., repeated blocks of values ​​within an image, or corresponding values ​​between successive images, optionally after taking into account motion between the images). Such approaches are found in a variety of well-known encoding techniques, such as gif, JPEG, and MPEG images and videos. Summary of the Invention [Problem to be solved by the invention]

[0004] However, providing more efficient image and video coding techniques remains a continuing challenge, which the present application seeks to overcome or alleviate. [Means for solving the problem]

[0005] In a first embodiment, according to claim 1, an image coding method is provided.

[0006] In a first embodiment, a method for style encoding is provided according to claim 12.

[0007] In another embodiment there is provided a method for image decoding according to claim 14.

[0008] In another embodiment there is provided a method for style decoding according to claim 16.

[0009] In another embodiment there is provided an image coding device according to claim 19.

[0010] In another embodiment, according to claim 20, there is provided a non-transitory machine-readable medium containing a representation of one or more images.

[0011] In another embodiment there is provided an image decoding device according to claim 21.

[0012] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary but are not restrictive of the invention. [Brief description of the drawings]

[0013] The present disclosure and many of the attendant advantages will become more readily understood from the following detailed description, when taken in conjunction with the accompanying drawings, in which: [Figure 1] 1 is a schematic diagram of an entertainment device that can be configured as an encoding device and / or a decoding device according to an embodiment; [Diagram 2] FIG. 2 is a schematic diagram of a neural network according to an embodiment. [Diagram 3] FIG. 2 is a schematic diagram of a neural network according to an embodiment. [Figure 4] 1 is a flowchart of an image encoding method according to an embodiment. [Diagram 5] 1 is a flowchart of an image decoding method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In the drawings, like reference numerals indicate the same or corresponding parts throughout the several views. In Figure 1, an example of an entertainment system 10 is a computer or console, such as the Sony PlayStation 5 (PS5).

[0015] The entertainment system 10 comprises a central processor 20. This may be a single-core or multi-core processor, consisting of, for example, eight cores, as in the PS5. The entertainment system also comprises a graphical processing unit (GPU) 30. The GPU may be physically separate from the CPU, or may be integrated with the CPU as a system-on-chip (SoC), as in the PS5.

[0016] The entertainment device also includes RAM 40. The CPU and GPU may each have separate RAM, or a shared RAM, as in the PS5. The RAM may also be physically separate, or integrated as part of the SoC, as in the PS5. Further storage is provided by a disk 50, either as an external or internal hard drive, an external solid state drive, or an internal solid state drive, as in the PS5.

[0017] The entertainment device may transmit and receive data via one or more data ports 60, such as a USB port, an Ethernet port, a Wi-Fi port, a Bluetooth port, etc., as appropriate, and may also optionally receive data via an optical drive 70.

[0018] Audio / visual output from an entertainment device is typically provided through one or more A / V ports 90 or through one or more wired or wireless data ports 60 .

[0019] An example of a device for displaying an image output by an entertainment system is a head mounted display "HMD" 120 such as a PlayStationVR2 (registered trademark) "PSVR2 (registered trademark)" worn by the user 1.

[0020] Where components are not integrated, they may be connected via dedicated data links or bus 100 as appropriate.

[0021] Interaction with the system is typically provided using one or more handheld controllers (130, 130A), such as the DualSense® controller (130) in the case of a PS5, and / or one or more VR controllers (130A-L, R) in the case of an HMD.

[0022] encoding device In embodiments herein, entertainment system 10 is an example of an encoding system operable to implement the methods and techniques described herein, which may be used to encode and / or decode images and / or video in accordance with the methods and techniques described herein.

[0023] In one embodiment herein, the encoding method includes training a machine learning system, such as a neural network.

[0024] An exemplary neural network is a fully connected multi-layer perceptron, i.e., a fully connected multi-layer network with one or more hidden layers. Typically, such neural networks operate as feed-forward neural networks that are trained using a backpropagation algorithm to approximate a target output in response to a corresponding input. More generally, neural networks can be implemented as fully connected neural networks, deep neural networks (DNNs), multi-layer perceptrons (MLPs), feed-forward artificial neural networks (ANNs), or convolutional neural networks (CNNs, or ConvNets).

[0025] In particular, in embodiments herein, each image / frame is encoded as a neural network, i.e., each neural network is trained to generate pixel data corresponding to an image, and the neural network is an implicit neural representation of the source content of that image / frame.

[0026] It is the sequences of the respective neural networks that are stored, transmitted or streamed instead of the image / frame data or their encoded versions.

[0027] This approach is highly efficient because the neural network can learn how to best represent a given image in its constituent weights. The bandwidth of the image can be controlled by the number of nodes in the neural network used to encode the image, which in turn determines the number of weights available to the neural network.

[0028] The degree of compression is therefore controlled by the number of nodes (usually hidden nodes) in the neural network used to represent each image. Each neural network learns a representation of the image that best suits the available bandwidth provided by the network weights.

[0029] 2, an exemplary neural network 200 includes a layer of input nodes 210, a layer of output nodes 240, and one or more hidden layers 220, 230. The hidden layers may be composed of any suitable number of nodes. Between each layer there is typically a fully connected set of weights 215, 225, 235. However, the network need not be so fully connected. Thus, some weights may not be included.

[0030] In the embodiments herein, two input nodes (x,y) indicate pixel locations in the image to be encoded, and three output nodes (R,G,B) indicate the color value of the pixel at the x,y pixel location. RGB is an exemplary color scheme; any suitable color (or grayscale) representation can be used, such as YUV or HSV.

[0031] The neural network is trained on a particular image (e.g., an image or an image frame of a video) using the RGB values ​​of the image at different x,y positions. Training can continue until an error criterion is met.

[0032] The error criteria consist of some or all of the following: i. The total error for all image pixels is below a threshold (indicating that the image accuracy meets a predefined quality / precision level). -Or the total error for a subset of pixels is below another threshold (simply indicating that a representative sample of pixels meets a predefined quality / accuracy level). ii. The average error over all pixels or sub-pixels is below a threshold (the average is a similar measure to the sum for the purposes of meeting a predefined quality / accuracy level). iii. N or more pixels have errors above a threshold (where N denotes the maximum allowable number of pixels with significant errors in the image). iv. The total error or average error has not changed by a threshold amount after M training iterations (i.e., the neural network has likely achieved the best results available in the current training process, or further training iterations have a sufficiently small return on the computational investment that they should not be continued).

[0033] Once trained, the neural network constructs an internal representation of the image in its weights.

[0034] Assuming the architecture / layout of the neural network is known and the decoder is known, then only the neural network weight values ​​can be sent to the decoder in a predefined order.

[0035] Optionally, during training, weight values ​​can be constrained to values ​​suitable for transmission. For example, typical weights are in the range 0 to 1. However, the system may be forced during training to use weight values ​​that are fractions of 256 or 65536 (e.g., by selecting the fraction closest to the calculated change in weight). Other bitwise value plantings may be considered, or one or more quantification schemes may be used to select the P values ​​that are most common within the network (or across each layer).

[0036] The above approaches are intended to use weight values ​​that can be efficiently transmitted to the decoder, ensuring that the network operates correctly / as expected.

[0037] Such an approach can also take into account changes in the training scheme. For example, the network can first be trained conventionally, and then the weight values ​​can be constrained to fractions as described above, or to values ​​corresponding to a quantization scheme, and then only a subset of the network (e.g., within one layer) can be given top-up training to compensate for the small changes in weight values ​​imposed on the entire network by this approach. Or, for example, most layers can be trained using fractions of 256, and one layer (e.g., the final set of weights leading to the output node) can be trained using fractions of 65535.

[0038] In either case, the neural network weights are transmitted from the encoding device (or a server or other sender that has access to the encoding device's stored results) to the decoding device, e.g., as a stream of data or as a file, or exported as part of a software product, such as on an optical disc.

[0039] Decoding Device The decoder queries the received neural network for each x,y pixel location and stores the output pixel information for that location.

[0040] It will be appreciated that for a typical HD image (1920x1080) this equates to over 2 million queries, and for a UHD image (3840x2160) this equates to over 8 million queries.

[0041] Thus, in an embodiment of the present disclosure, the decoding device includes a graphics processing unit with multiple parallel shaders, each operable to query a respective copy of the neural network to generate results in parallel, thus speeding up the decoding process.

[0042] In this way, rather than decoding the image in a traditional way, the decoder prompts one or more copies of the neural network to generate image information at each pixel location.

[0043] Variations I and P networks The above approach provides a means to encode an image or an image sequence, such as a video, by creating a respective trained neural network for each image that outputs pixel information corresponding to the image for each queried pixel location.

[0044] This requires the neural network to build a substantially complete internal representation of the image. As such, the neural network can be thought of as an "I-frame" neural network, i.e., one that completely encapsulates the image it represents. While this neural network can perform very efficiently, there is room to further improve it for a sequence of images.

[0045] Referring now to Figure 3, this shows a modified neural network 300 that, like Figure 2, includes an input layer 310, an output layer 340, and one or more hidden layers 320, 330. However, in this case there are three additional inputs, Rt-1, Gt-1, Bt-1, which correspond to the RGB values ​​of the x,y locations of the previous image in the image sequence. More specifically, the additional inputs correspond to the RGB values ​​of the x,y locations in the image generated by the previous neural network, as these are accessible to the decoding device when prompting the subsequent neural network to generate an output.

[0046] This modified neural network is then trained to output pixel values ​​Rt, Gt, Bt for the current image based on the pixel location and the pixel value at that location in the previous image.

[0047] It takes advantage of the assumption that previous images in an image sequence are usually very similar to the current image. As a result, in this case the neural network only needs to learn how to modify the inputs from previously decoded / output images to generate a new image, without having to store a complete internal representation of the image itself. As a result, the neural network is generally able to obtain results of similar quality with far fewer weights, resulting in a smaller overall size of the neural network. As a result, the bandwidth required to transmit or store the neural network is also smaller.

[0048] Such a neural network can be thought of as a P-frame neural network, i.e., a neural network that relies on ongoing image data to generate an output.

[0049] Optionally, the three additional inputs may be RGB values ​​of different pixel locations in a previous image relative to the pixel location represented by the input x,y coordinates in the current image being encoded. In particular, the input RGB values ​​may take into account motion between frames and modify the RGB values ​​at the location in the proceeding image where the corresponding image element was at that time to better match the current RGB values ​​in x,y coordinates. The motion between frames may be limited to panning across the image, or it may be more complex motion within the image, for example as represented by a motion vector.

[0050] In order for the same RGB values ​​to also be input to the decoder, the decoder must either independently estimate the same inter-frame motion or provide the inter-frame motion information as separate data associated with the neural network when it is transmitted, or the neural network can be trained to output such information as well, by providing it as target data during training.

[0051] Thus, the sequence of trained neural networks may consist of a mixture of I-frame neural networks and P-frame neural networks. Typically, I-frames are used when there is a scene change or when the difference between subsequent images is too large for the P-frame neural networks to produce an output with an acceptable level of error. P-frames may optionally be accompanied by information indicative of the motion between frames or to allow for computation at the decoder.

[0052] It is also possible to improve the efficiency of the I-frame neural network by giving the Rt-1, Gt-1, Bt-1 inputs to the neural network but using a selected standard image. For example, the standardized prior image may be one of Q images, each being the average of a particular type of image, e.g. landscape images, streetscape images, talking head images, etc. These may contain typical features, although they cannot be described in detail, such as a landscape image having a bright upper half of the image with a particular color range and a darker lower half with a different particular color range, a streetscape image having a different division of light and color, a talking head image having a general human outline and a general background, etc. In this case, Q may be, for example, 8, 16, 32, etc.

[0053] This neural network typically requires more weights than a regular P-frame neural network, since the standardized common precursor image is only a poor approximation of the current image, but because it only computes the difference, it requires fewer weights than a pure I-frame network that has no input from the common precursor image, allowing it to code images more efficiently.

[0054] As a result, during training, the standardized prior image in the set of Q images that is closest to the image being encoded (e.g., in terms of pixel value squared error) is selected and used to provide the Rt-1, Gt-1, Bt-1 inputs during training. The selected image is indicated as metadata along with the trained network, for example, when it is sent to the decoder. The decoder can select the correct standardized prior image when prompting for output from the trained network.

[0055] Q-standard images may also be generated from the specific content being encoded, where the colors and lighting in the standard images provide a better initial match than if a completely generic image were used. These Q-standard images are then encoded using an I-frame neural network or any other conventional technique and sent, for example, at the start of the stream or at any event before they are needed.

[0056] In this way, the inputs and outputs of each neural network, and potentially their footprint (number of weights), can be made the same, at least for the majority of the encoded stream. This can make the decoding process simpler in terms of consistently placing and running neural networks on the GPU.

[0057] Block-wise input and / or output This means that internally there are multiple weights for every pixel, because the neural network is likely to be much larger in practice than the original image, since many of the weights contributing to individual pixels are highly redundant.

[0058] However, outputting RGB (or other color scheme) pixel values ​​for a single pixel represents the opposite extreme: it allows for efficient use of weights, especially for pixels that connect to the output layer. As a result, these rates can be very sensitive to changes in values ​​or limitations in value choices (such as those mentioned above to transmit weight values ​​more efficiently). It also forces the decoder to implement a large number of queries to generate the entire image.

[0059] Thus, optionally, the neural network may be configured to provide pixel values ​​for a larger subset of the image. For example, the neural network may output pixel values ​​for 2x2, 4x4, 8x8, 16x16, 32x32, or 64x64 pixel blocks (e.g., macroblocks or equivalent). Thus, the output layer (240, 340) may contain more nodes than shown in Figure 2 or Figure 3. Also, the hidden layer (or the hidden layer next to the output layer 230, 330) may contain fewer nodes, and the number of weights (235, 335) between this hidden layer and the output layer may be approximately similar to that of the single pixel output version.

[0060] Similarly, optionally, and independent of the choice of single or multiple output pixels, the neural network may be configured to have many more input values ​​than just the RGB values ​​of a single pixel at, for example, an x,y location. For example, the neural network may have 2x2, 3x3, 4x4, 5x5, 8x8, 16x16, 32x32, or 64x64 pixel blocks of input pixel values. Again, the hidden layers (220, 320) adjacent to the input layers (210, 310) may be configured with fewer codes, such that the overall number of intervening weights (215, 315) is roughly similar to the single-input version.

[0061] Optionally, the input RGB values ​​may correspond to a lower resolution version of the target image, so for example a 2x2 or 4x4 set of RGB values ​​may represent a downsampled 4x4, 8x8 or 16x16 pixel area of ​​the prior image.

[0062] If the input region of the preceding image is larger than the output region of the respective image, the input region may provide some context to the neural network in generating the output region values, particularly aiding in some styles (described elsewhere herein). However, some of this may be redundant, which may improve image quality but may reduce the overall efficiency of the network. On the other hand, if the input region is the same size as the output region (whether using a downsampled version of the input region or per-pixel values ​​of the input region), the network operates in a similar manner as before, but for multiple pixels in a block.

[0063] Also, in neural networks that support blocks of pixel data, the x, y inputs correspond to block positions rather than pixel positions.

[0064] Stereoscopy The neural network can also be trained to output two sets of RGB values ​​for corresponding coordinates in a stereo image pair. In this case (or more generally, where applicable whether stereoscopic or not), the neural network may also receive depth information (e.g., from a corresponding depth map) or disparity information to aid in the internal representation of the stereoscopic scene. In this case, due to the disparity / offset in the stereoscopic images, the same location in each image is likely to contain different elements, and therefore the two sets of RGB values ​​are likely to be different.

[0065] Thus, the neural network is trained to output two sets of RGB values ​​and disparity / offset values. The first set of RGB values ​​corresponds to the input coordinates of one (e.g., left) image of the image pair. The second set of RGB values ​​corresponds to a position offset from the input coordinates according to the stereo disparity of the other (e.g., right) image of the image pair. In this way, the RGB values ​​are likely to be very similar, since the neural network can learn to output the same elements of the scene as stereo offsets.

[0066] These two approaches represent different internal representation overheads for the process of predicting the colors and positions of stereoscopic image elements. Different approaches may be more efficient for different types of content or within specific content.

[0067] Multiple Styles Different users may want to see content in different ways. This may be for purely aesthetic reasons, or it may be beneficial to certain users. For example, it may be advantageous to provide content in a format that is optimal for color-blind users (e.g. varying degrees of red-green color deficiency, or similarly varying degrees of blue-yellow color deficiency).

[0068] Aesthetic alterations may include differences in contrast and / or color balance, or other effects such as creating a painted, pastel, or line drawing version of the image, or other texture alterations, highly saturated versions, warmer or cooler color temperature versions, etc.

[0069] Traditionally, such modifications are applied by the client device after the source image is decoded, but in embodiments herein, the style can optionally be incorporated into the neural network rather than being applied at the decoding device.

[0070] Optionally, if a style is expected to persist for a significant portion of the viewing period (e.g., as may be the case for a color-blind user, or where a user selects a particular color temperature version in response to television settings or characteristics of the local environment), a neural network (in any of the forms previously described herein) may be specially trained to output pixel values ​​in that style, and the associated sequence of the neural network may be streamed.

[0071] However, for example in a broadcast scenario where a single stream may be received by multiple viewers, it may not be practical to output individually tailored streams due to the different needs of the viewers.

[0072] In this case, optionally the neural network (which may take any of the forms previously described herein, e.g., fully encapsulating the encoded image, modifying a standardized image, outputting for a single pixel at a time, modifying a prior image whether outputting for multiple pixels at a time, etc.) may also have one or more style inputs.

[0073] The style input may take the form of a binary input or a set of flags (e.g., a one-hot vector) to each additional input node of the network. The binary or one-hot flag pattern indicates a style choice. Typically, one style represents neutral, i.e., the original image. Alternatively or additionally, the style can be represented as a vector, a scalar, or indeed text (e.g., by converting the text to a vector and optionally abstracting the text).

[0074] The neural network is then trained on multiple versions of the image (i.e. at least two), one for each supported style. Thus, in this case, the neural network may output different RGB values ​​for the same pixel location depending on the style indicated. In this case, for comparable image quality / precision, the neural network will be slightly larger than a neural network that does not encode a single style or any style at all (i.e. a "neutral" style). Such a neural network can be thought of as a combination of an I or P frame network with a further P frame network for each supported style. The weights driven by the style input then interact with other weights to modify the image produced by the rest of the network, which is responsive to the input coordinates and, optionally, the input RGB values.

[0075] This multiple style version of the neural network is then sent to a decoding device in a similar manner as described elsewhere herein, where the decoding device may be prompted to output pixel values ​​in a similar manner, but in this case an associated style selected by a user (e.g., via a UI) may also be input by the decoding device to the network so that RGB values ​​appropriate for that style are output.

[0076] Thus, in this approach, rather than styles being applied to an image decoded on a client device, multiple styles are embodied in a neural network representation of an image (or modifications of the image), and the neural network is prompted by the decoder to output the associated style type.

[0077] Parameterizing Styles As previously described herein, when a neural network is trained with one or more styles, optionally, the one or more styles may include one or more parameter inputs for controlling one or more aspects of the style.

[0078] For example, a style that changes the color temperature of an image may include a time of day parameter, such that the color temperature gradually changes from cool to warm later in the evening to aid in user sleep, etc. In this case, a neural network may be trained with different color temperatures associated with different time of day parameters.

[0079] Alternatively, the neural network may be trained on warm and cool versions of the image as separate styles. In this case, the parameter may be a value representing the relative bias between the two styles. In this case, for example, the style inputs, which together represent a potential range or spectrum of the output, may each be continuous values ​​rather than flags. In this case, each value may indicate an internet contribution to the output.

[0080] Similarly, each style may include parameter inputs that indicate the degree / strength of stylization relative to a neutral version of the image, or a continuous style input, where again the neural network is trained for different parameter input values ​​and different strengths / degrees of stylization in the target output.

[0081] It is also possible to combine different styles by providing a style input indicating the degree / strength to which each style is applied: for example, one could apply a pencil sketch style and another a sepia color palette style to create a composite effect that is not itself included in the training set of target images.

[0082] Post-processing styles Given that the output of a neural network according to the techniques herein is an RGB value (or similar pixel value) for a given input coordinate, optionally a stream according to embodiments herein may include a neural network with an unstyled / neutral style output and a separate second neural network that applies one or more styles to the output of the first neural network.

[0083] This second neural network can be trained in a similar manner to the style-based neural network described earlier in this specification: it operates like the P-frame network, receiving RGB values ​​and at least a first style value, and typically coordinate values, from the first neural network, and outputs RGB values ​​corresponding to the selected style.

[0084] In this case, the first neural network can be treated as a baseline common transmission to the clients, while the second neural network can be treated as an enhancement layer that is fed to clients requesting modifications to the image (i.e., a different style).

[0085] Other post-processing Regardless of whether the style is generated by or applied to the output of the first neural network, optionally other more general post-processing may be performed on the resulting image.

[0086] Such post-processing may include error masking, upscaling, jitter reduction, etc.

[0087] summary Referring now to FIG. 4, in a summary embodiment herein, a method for image encoding (eg, for still or video images) comprises the following steps.

[0088] In a first step S410, each neural network is trained to output pixel data for each image, as described elsewhere herein.

[0089] In a second step S420, the trained neural network is output as a coded representation of the image (whether as a single image or as a stream of continuous images such as a video, broadcast, and / or file), as described elsewhere herein.

[0090] The present invention is not limited to the specific configuration and operation described and claimed below, and it is understood that variations thereon are also within the scope of the present invention.

[0091] The neural network may include an input for receiving a location coordinate of the image, and an output providing a color value of the pixel at the location coordinate.

[0092] In this case, the neural network may further include an input for receiving a color value of a pixel in the prompt image corresponding to the received position coordinates.

[0093] In this case, the prompt image is i. the image immediately preceding the image, or ii. An image selected from a set of common images as the image most similar to the image may be also possible.

[0094] In this case, the color value received as the color value of a pixel in the prompt image may be the color value of the pixel offset from the received position coordinates by an amount corresponding to the inter-image motion between the image and the prompt image.

[0095] In this case, the neural network may include an input for receiving a location coordinate of the image, and an output for providing a color value of a group of pixels at the received location coordinate, where the group may be smaller than the entire image.

[0096] In this case, the neural network may include inputs for receiving image position coordinates and outputs for providing color values ​​for each pixel of the stereo image pair.

[0097] The neural network may include inputs defining two or more styles, and the neural network may be trained to output pixel data of the image in two or more styles, and at least one of the styles may modify the output pixel data of the image.

[0098] In this case, the style is i. color characteristics related to the type of display on which the image is displayed (e.g., to account for differences in color balance or brightness between displays or types of displays, such as televisions versus head mounted displays); ii. Color characteristics that correspond to the time of day (e.g., lowering color temperature in the evening to help the viewer sleep) iii. Color characteristics that address a viewer's medical condition (e.g., to assist users who are color-blind or who benefit from high-contrast images); iv. Aesthetic color characteristics (e.g., sepia or highly saturated 1950s style film effects), or v. Texture characteristics (e.g., to make an image look painted or drawn, or to have a grainy film quality, or to have a raster-by-cathode ray style) The output pixel data may be modified with respect to

[0099] In this case, alternatively or additionally, the neural network may include inputs for one or more styles indicating the degree or strength to which the style should be applied to the output pixel data (e.g., alone or in combination with one or more other styles supported by the neural network).

[0100] Instead of incorporating style into a single neural network for the image, the method may include training a further neural network to output style-modified pixel data for the image and outputting the trained further neural network as a complementary coded representation of the image, where the input to the further neural network may include the output pixel data of the neural network.

[0101] The above may involve coding the image with a neural network, or optionally coding the image in two or more additional styles (one of which may be the "original"), or optionally providing a second neural network to apply the two or more additional styles, while optionally coding only the style, independently of the method of image coding or decoding.

[0102] That is, one embodiment of the present disclosure is a style encoding method, comprising the steps of providing a trained neural network and outputting the trained neural network to a decoding device for use with a decoded image, the trained neural network receiving color values ​​and including inputs for defining two or more styles for the image and outputs for providing color values, the trained neural network being trained to output colors for the image in two or more styles, at least one of the styles modifying the output colors in response to the input color values.

[0103] In this case, the trained neural network may include an input for receiving position coordinates of the image, and the output may be trained to provide a color value of the pixel at the position coordinates.

[0104] In this way, one can transmit a style network that can apply style to an existing image (e.g., an image decoded from another network as described elsewhere in this specification, or an image decoded using any other technique).

[0105] The decoding device may apply the indicated style to the image universally or in response to a current pixel location. Accordingly, one embodiment of the present specification is a style decoding method, comprising the steps of receiving a neural network trained to output color values ​​in response to receiving color values ​​and input style values ​​defining two or more styles, inputting color values ​​of an image and at least a first style value to the trained neural network, receiving output color values ​​from the trained neural network, and displaying the image using the output color values ​​instead of the input color values, where at least one of the styles modifies the output color in response to the input color values.

[0106] The trained neural network may include an input for receiving position coordinates of an image, and an output may be trained to provide a color value of a pixel at said position coordinates.

[0107] Referring now to FIG. 5, a summary embodiment of the present specification is an image decoding method including the following steps.

[0108] The method includes a step S510 of receiving a neural network trained to output pixel data for an image, and a step S520 of prompting the trained neural network to output the pixel data for the image.

[0109] The present invention is not limited to the specific configuration and operation described and claimed below, and it is understood that variations thereon are also within the scope of the present invention.

[0110] The trained neural network may include an input for receiving position coordinates of the image and an output providing a color value of the pixel at the position coordinate, and prompting the trained neural network may include inputting all position coordinates in the image and matching the outputs.

[0111] Modifications of the decoding device corresponding to modifications described elsewhere herein will be apparent to those skilled in the art. Such modifications include, for example: generating output on a pixel-by-pixel or pixel-group basis; using color values ​​from a prompt image, which may be from the ongoing generated image or a best-matching generic image; outputting a pixel from each image of a stereo pair; inputting a style selection or selections, or parametric values ​​associated with one or more style selections; receiving a first neural network that encodes an image and a second neural network that encodes style variations to generate a style-restyled version of the image; performing these operations in parallel on a GPU or equivalent processor having parallel processing capabilities, such that prompts of each instance of the neural network to generate individual pixels or groups of pixels as output can be performed in parallel to speed up the overall generation of the image.

[0112] The above methods may be implemented on suitably adapted conventional hardware, either by means of software instructions, or by the incorporation or substitution of dedicated hardware.

[0113] Thus, the necessary adaptation to existing portions of the prior equivalent device may be implemented in the form of a computer program product including processor-implementable instructions stored on a non-transitory machine-readable medium, such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory, or any combination of these or other storage media, or may be realized in hardware as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) or other configurable circuitry suitable for use in adapting the prior equivalent device. Alternatively, such computer program may be transmitted via data signals over a network, such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

[0114] Thus, in a general embodiment of the present specification, an image encoding device, eg a server (not shown) or an entertainment device 10 such as a Sony PlayStation®, may be configured as follows.

[0115] First, a training processor (e.g., GPU 30, optionally in conjunction with CPU 20) configured (e.g., by suitable software instructions) for each image to train a respective neural network to output pixel data for that image, as described elsewhere herein.

[0116] Second, as described elsewhere herein, an output processor (e.g., CPU 20, optionally in conjunction with GPU 30) configured (e.g., by appropriate software instructions) to output the trained neural network as a coded representation of an image (e.g., an image frame, either as a stand-alone image or as part of a sequence such as a video).

[0117] Specific examples that implement the methods and techniques described herein (eg, by using appropriate software instructions) are contemplated within the scope of this application.

[0118] The resulting output may be a non-transitory, machine-readable medium (such as an HDD or SSD storage, or an optical drive storage) that includes representations of one or more images formed by a respective neural network for the one or more images, each trained to output pixel data for a respective image (e.g., a stand-alone image or an image frame as part of a sequence such as a video), as described elsewhere herein.

[0119] This output may be distributed or broadcast or streamed on a medium as appropriate.

[0120] Meanwhile, in another summary embodiment of the present specification, an image decoding device, for example a server (not shown) or an entertainment device 10 such as a Sony PlayStation 5 (registered trademark), is configured as follows.

[0121] First, a receiver (e.g., CPU 20, optionally in conjunction with GPU 30) configured (e.g., by appropriate software instructions) to receive (via data port 60, SSD 50, optical drive 70, or A / V port 90) each neural network trained to output pixel data for an image, as described elsewhere herein.

[0122] Second, an image generation processor (e.g., GPU 30, optionally in conjunction with CPU 20) configured (e.g., by appropriate software instructions) to prompt the neural network to output pixel data for that image, as described elsewhere herein.

[0123] Again, specific examples of implementing the methods and techniques described herein (e.g., through the use of appropriate software instructions) are contemplated within the scope of this application, including, but not limited to, the following:

[0124] The image generation processor is a graphics processing unit having multiple parallel processing units that outputs pixel data for an image in parallel to parallel instances of the neural network.

[0125] The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. Accordingly, the disclosure of the present invention, as well as the remaining claims, are intended to be illustrative and not limiting of the scope of the present invention. The present disclosure, including readily identifiable variations of the teachings herein, defines in part the scope of the preceding claim terms, so as not to dedicate the inventive subject matter to the public.

Claims

1. 1. An image encoding method, comprising: training a neural network to output pixel data relating to an image; outputting the trained neural network as a coded representation of the image; The method according to claim 1, further comprising:

2. The neural network comprises: an input for receiving position coordinates of the image; an output providing a color value of the pixel at said location coordinate; 2. The method of claim 1, comprising:

3. 3. The method of claim 2, wherein the neural network further includes an input for receiving a color value of a pixel in a prompt image that corresponds to the received position coordinates.

4. The prompt image is i. the image immediately preceding said image, or ii. an image selected from a set of common images as the image most similar to said image; The method according to claim 3, characterized in that

5. 4. The method of claim 3, wherein the color values ​​received for the pixels of the prompt image are pixel color values ​​offset from the received position coordinates by an amount corresponding to inter-image motion between the image and the prompt image.

6. The neural network comprises: an input for receiving position coordinates of said image; an output providing a color value of the group of pixels at the received position coordinates; Including, A method according to any of claims 2 to 5, wherein the group is smaller than the entire image.

7. The neural network comprises: an input for receiving position coordinates of said image; an output providing a color value for each pixel of the stereo image pair; 6. The method according to claim 2, further comprising:

8. the neural network includes inputs defining two or more styles; the neural network is trained to output pixel data of the image in the two or more styles; 8. A method according to any preceding claim, wherein at least one of the styles modifies output pixel data of the image.

9. The style is: i. color characteristics associated with the type of display on which the image is displayed; ii. Color characteristics corresponding to time of day; iii. Color characteristics corresponding to a viewer's medical condition; iv. Aesthetic color properties, or v. Texture characteristics 9. The method of claim 8, further comprising modifying the output pixel data with respect to:

10. 10. A method according to claim 8 or 9, wherein the neural network includes inputs relating to one or more styles indicating the degree or strength to which the style should be applied to output pixel data.

11. training a further neural network to output restyled pixel data for said image; outputting the trained further neural network as a complementary coded representation of the image; Including, 8. A method according to any preceding claim, wherein the input to the further neural network comprises output pixel data of the further neural network.

12. A style encoding method comprising the steps of: providing a trained neural network; outputting the trained neural network to a decoding device for use in decoding an image; Including, The trained neural network comprises: an input for receiving color values ​​and defining two or more styles for the image; an output for providing said color value; Including, the trained neural network is trained to output colors for the image in two or more styles; A method according to claim 1, wherein at least one of said styles modifies output colors depending on said input color values.

13. the trained neural network includes an input for receiving position coordinates of the image; 13. The method of claim 12, wherein the output is trained to provide a color value of a pixel at the location coordinate.

14. An image decoding method, comprising: receiving a neural network trained to output pixel data relating to an image; prompting the trained neural network to output pixel data of the image; The method according to claim 1, further comprising:

15. The trained neural network comprises: an input for receiving position coordinates of the image; an output providing a color value of the pixel at said location coordinate; Including, The step of prompting the trained neural network comprises: inputting all position coordinates in the image; matching the outputs; 15. The method of claim 14, comprising:

16. A style decoding method, comprising the steps of: receiving a neural network trained to output a color value in response to receiving a color value and input style values ​​defining two or more styles; inputting color values ​​and at least a first style value of an image into the trained neural network; receiving color values ​​output from the trained neural network; displaying the image using the output color values ​​instead of the input color values; Including, A method according to claim 1, wherein at least one of said styles modifies output colors depending on said input color values.

17. the trained neural network includes an input for receiving position coordinates of the image; 17. The method of claim 16, wherein the output is trained to provide a color value of a pixel at the location coordinate.

18. A computer program comprising computer executable instructions adapted to cause a computer system to carry out a method according to any one of claims 1 to 17.

19. a training processor for training the neural network to output pixel data for an image; an output processor for outputting the trained neural network as a coded representation of the image; An image encoding device comprising:

20. a neural network-generated image representation of the image; The neural network is trained to output pixel data for the image.

21. a receiver for receiving the trained neural network to output pixel data relating to an image; an image generation processor that prompts the trained neural network to output pixel data of the image; An image decoding device comprising:

22. 22. The image decoding apparatus of claim 21, wherein the image generation processor is a graphics processing unit having multiple parallel processing units and outputs pixel data for the image in parallel to parallel instances of the neural network.

Images